[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880383132139-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:46:23 dist-test-slave-2sb1 krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:46:23 dist-test-slave-2sb1 krb5kdc[399](info): set up 2 sockets
Jun 19 14:46:23 dist-test-slave-2sb1 krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260619 14:46:24.267254 392 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.003s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880383132139-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880383132139-392-0/krb5kdc/test-user.keytab.
Jun 19 14:46:24 dist-test-slave-2sb1 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880384, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:46:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:46:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 14:46:24.420153 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880383132139-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880383132139-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880383132139-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43775
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43775
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:46:24.550590 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:46:24.550933 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:46:24.550977 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:46:24.556209 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:46:24.556324 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:46:24.556347 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:46:24.556367 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:46:24.556385 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:46:24.562776 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880383132139-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880383132139-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43775
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880383132139-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:46:24.563931 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:46:24.565052 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 14:46:24.576467 412 server_base.cc:1034] running on GCE node
W20260619 14:46:24.576529 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:24.576831 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:24.576872 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:24.577565 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:46:24.578832 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:46:24.582185 412 hybrid_clock.cc:648] HybridClock initialized: now 1781880384582077 us; error 111 us; skew 500 ppm
Jun 19 14:46:24 dist-test-slave-2sb1 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880384, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:46:24.587052 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 14:46:24.588863 412 webserver.cc:460] Webserver started at http://127.0.98.62:45517/ using document root <none> and password file <none>
I20260619 14:46:24.590102 412 fs_manager.cc:359] Metadata directory not provided
I20260619 14:46:24.590215 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:46:24.590553 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:46:24.595847 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880383132139-392-0/minicluster-data/master-0/data/instance:
uuid: "22ac4e36bdb04677bfe7b3f3982f221d"
format_stamp: "Formatted at 2026-06-19 14:46:24 on dist-test-slave-2sb1"
server_key: "e7fb0bc7a51fb47a327bcc1b6ead125e"
server_key_iv: "95a17a4092038e6ff152ef479e7b2c8b"
server_key_version: "encryptionkey@0"
I20260619 14:46:24.596796 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880383132139-392-0/minicluster-data/master-0/wal/instance:
uuid: "22ac4e36bdb04677bfe7b3f3982f221d"
format_stamp: "Formatted at 2026-06-19 14:46:24 on dist-test-slave-2sb1"
server_key: "e7fb0bc7a51fb47a327bcc1b6ead125e"
server_key_iv: "95a17a4092038e6ff152ef479e7b2c8b"
server_key_version: "encryptionkey@0"
I20260619 14:46:24.602123 412 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 14:46:24.605990 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:24.607743 412 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260619 14:46:24.607889 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880383132139-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880383132139-392-0/minicluster-data/master-0/wal
uuid: "22ac4e36bdb04677bfe7b3f3982f221d"
format_stamp: "Formatted at 2026-06-19 14:46:24 on dist-test-slave-2sb1"
server_key: "e7fb0bc7a51fb47a327bcc1b6ead125e"
server_key_iv: "95a17a4092038e6ff152ef479e7b2c8b"
server_key_version: "encryptionkey@0"
I20260619 14:46:24.608090 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880383132139-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880383132139-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880383132139-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:46:24.622530 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:46:24.626235 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:46:24.626513 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:46:24.635070 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43775
I20260619 14:46:24.635694 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43775 every 8 connection(s)
I20260619 14:46:24.636236 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:46:24.639775 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:46:24.639837 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 412
I20260619 14:46:24.639987 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880383132139-392-0/minicluster-data/master-0/wal/instance
I20260619 14:46:24.641731 392 external_mini_cluster.cc:1428] Setting key cdd121ed8f359e501851e63144873874
I20260619 14:46:24.648217 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22ac4e36bdb04677bfe7b3f3982f221d: Bootstrap starting.
I20260619 14:46:24.652088 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 22ac4e36bdb04677bfe7b3f3982f221d: Neither blocks nor log segments found. Creating new log.
I20260619 14:46:24.653159 480 log.cc:824] T 00000000000000000000000000000000 P 22ac4e36bdb04677bfe7b3f3982f221d: Log is configured to *not* fsync() on all Append() calls
I20260619 14:46:24.655869 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22ac4e36bdb04677bfe7b3f3982f221d: No bootstrap required, opened a new log
I20260619 14:46:24.660336 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 22ac4e36bdb04677bfe7b3f3982f221d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22ac4e36bdb04677bfe7b3f3982f221d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43775 } }
I20260619 14:46:24.660705 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P 22ac4e36bdb04677bfe7b3f3982f221d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:46:24.660790 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 22ac4e36bdb04677bfe7b3f3982f221d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22ac4e36bdb04677bfe7b3f3982f221d, State: Initialized, Role: FOLLOWER
I20260619 14:46:24.661276 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22ac4e36bdb04677bfe7b3f3982f221d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22ac4e36bdb04677bfe7b3f3982f221d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43775 } }
I20260619 14:46:24.661417 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P 22ac4e36bdb04677bfe7b3f3982f221d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:46:24.661509 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P 22ac4e36bdb04677bfe7b3f3982f221d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:46:24.661614 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 22ac4e36bdb04677bfe7b3f3982f221d [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:46:24.662866 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P 22ac4e36bdb04677bfe7b3f3982f221d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22ac4e36bdb04677bfe7b3f3982f221d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43775 } }
I20260619 14:46:24.663272 480 leader_election.cc:302] T 00000000000000000000000000000000 P 22ac4e36bdb04677bfe7b3f3982f221d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22ac4e36bdb04677bfe7b3f3982f221d; no voters:
I20260619 14:46:24.663621 480 leader_election.cc:288] T 00000000000000000000000000000000 P 22ac4e36bdb04677bfe7b3f3982f221d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:46:24.663798 485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 22ac4e36bdb04677bfe7b3f3982f221d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:46:24.664172 485 raft_consensus.cc:686] T 00000000000000000000000000000000 P 22ac4e36bdb04677bfe7b3f3982f221d [term 1 LEADER]: Becoming Leader. State: Replica: 22ac4e36bdb04677bfe7b3f3982f221d, State: Running, Role: LEADER
I20260619 14:46:24.664633 485 consensus_queue.cc:237] T 00000000000000000000000000000000 P 22ac4e36bdb04677bfe7b3f3982f221d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22ac4e36bdb04677bfe7b3f3982f221d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43775 } }
Jun 19 14:46:24 dist-test-slave-2sb1 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880384, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:46:24.665081 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 22ac4e36bdb04677bfe7b3f3982f221d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:46:24.666591 487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22ac4e36bdb04677bfe7b3f3982f221d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 22ac4e36bdb04677bfe7b3f3982f221d. Latest consensus state: current_term: 1 leader_uuid: "22ac4e36bdb04677bfe7b3f3982f221d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22ac4e36bdb04677bfe7b3f3982f221d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43775 } } }
I20260619 14:46:24.666730 487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22ac4e36bdb04677bfe7b3f3982f221d [sys.catalog]: This master's current role is: LEADER
I20260619 14:46:24.667096 486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22ac4e36bdb04677bfe7b3f3982f221d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "22ac4e36bdb04677bfe7b3f3982f221d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22ac4e36bdb04677bfe7b3f3982f221d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43775 } } }
I20260619 14:46:24.667184 486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22ac4e36bdb04677bfe7b3f3982f221d [sys.catalog]: This master's current role is: LEADER
W20260619 14:46:24.670594 500 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 22ac4e36bdb04677bfe7b3f3982f221d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:46:24.670691 500 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:46:24.670786 496 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:46:24.671582 496 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:46:24.672842 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:24.646325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52360 (local address 127.0.98.62:43775)
0619 14:46:24.647044 (+ 719us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:24.647073 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:24.648891 (+ 1818us) server_negotiation.cc:407] Connection header received
0619 14:46:24.652794 (+ 3903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:24.652820 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:24.653302 (+ 482us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:24.653748 (+ 446us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:24.656390 (+ 2642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:24.657980 (+ 1590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:24.659586 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:24.660284 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:24.660477 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:24.666175 (+ 5698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:24.666183 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:24.666202 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:24.666243 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:24.668639 (+ 2396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:24.669523 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:24.669531 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:24.669536 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:24.669691 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:24.670155 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:24.670160 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:24.670164 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:24.670694 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:24.670886 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:24.671527 (+ 641us) server_negotiation.cc:299] Negotiation successful
0619 14:46:24.671810 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":304,"threads_started":1}
I20260619 14:46:24.678056 496 catalog_manager.cc:1347] Generated new cluster ID: 5cdae5cfac354e56a27c4688c69c2882
I20260619 14:46:24.678148 496 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:46:24.717602 496 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:46:24.718852 496 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:46:24.733253 496 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 22ac4e36bdb04677bfe7b3f3982f221d: Generated new TSK 0
I20260619 14:46:24.734200 496 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880383132139-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880383132139-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 14:46:24.776563 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880383132139-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880383132139-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880383132139-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880383132139-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43775
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880383132139-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:46:24.899896 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:46:24.900369 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:46:24.900471 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:46:24.908373 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:46:24.908485 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:46:24.908629 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 14:46:24.918170 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880383132139-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880383132139-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880383132139-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880383132139-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43775
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880383132139-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:46:24.919754 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:46:24.921432 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:46:24.935514 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:24.935514 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:24.936731 506 server_base.cc:1034] running on GCE node
W20260619 14:46:24.938086 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:24.938892 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:46:24.940060 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:46:24.941303 506 hybrid_clock.cc:648] HybridClock initialized: now 1781880384941276 us; error 42 us; skew 500 ppm
Jun 19 14:46:24 dist-test-slave-2sb1 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880384, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:46:24.945937 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 14:46:24.947476 506 webserver.cc:460] Webserver started at http://127.0.98.1:40867/ using document root <none> and password file <none>
I20260619 14:46:24.948110 506 fs_manager.cc:359] Metadata directory not provided
I20260619 14:46:24.948179 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:46:24.948370 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:46:24.950873 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880383132139-392-0/minicluster-data/ts-0/data/instance:
uuid: "37cb7b66f3f241ec9defc52eb2dc3d43"
format_stamp: "Formatted at 2026-06-19 14:46:24 on dist-test-slave-2sb1"
server_key: "68739271ad282c1bf9fa50d4f7b8a765"
server_key_iv: "817badb9095d3087b4b6b65255e874d1"
server_key_version: "encryptionkey@0"
I20260619 14:46:24.951573 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880383132139-392-0/minicluster-data/ts-0/wal/instance:
uuid: "37cb7b66f3f241ec9defc52eb2dc3d43"
format_stamp: "Formatted at 2026-06-19 14:46:24 on dist-test-slave-2sb1"
server_key: "68739271ad282c1bf9fa50d4f7b8a765"
server_key_iv: "817badb9095d3087b4b6b65255e874d1"
server_key_version: "encryptionkey@0"
I20260619 14:46:24.956468 506 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260619 14:46:24.959729 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:24.961738 506 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:46:24.961938 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880383132139-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880383132139-392-0/minicluster-data/ts-0/wal
uuid: "37cb7b66f3f241ec9defc52eb2dc3d43"
format_stamp: "Formatted at 2026-06-19 14:46:24 on dist-test-slave-2sb1"
server_key: "68739271ad282c1bf9fa50d4f7b8a765"
server_key_iv: "817badb9095d3087b4b6b65255e874d1"
server_key_version: "encryptionkey@0"
I20260619 14:46:24.962091 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880383132139-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880383132139-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880383132139-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:46:24.977144 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:46:24.981032 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:46:24.981261 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:46:24.981901 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:46:24.982813 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:46:24.982900 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:24.982977 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:46:24.983021 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:24.994007 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38119
I20260619 14:46:24.994055 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38119 every 8 connection(s)
I20260619 14:46:24.995082 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880383132139-392-0/minicluster-data/ts-0/data/info.pb
I20260619 14:46:24.995563 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 506
I20260619 14:46:24.995726 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880383132139-392-0/minicluster-data/ts-0/wal/instance
I20260619 14:46:24.996027 392 external_mini_cluster.cc:1428] Setting key 4259b85b87020631d3d07afedd928d4f
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Jun 19 14:46:25 dist-test-slave-2sb1 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880384, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:46:25.016531 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:24.998072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52711 (local address 127.0.98.62:43775)
0619 14:46:24.998210 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:24.998213 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:24.999323 (+ 1110us) server_negotiation.cc:407] Connection header received
0619 14:46:25.001046 (+ 1723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:25.001049 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:25.001245 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:25.001388 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:25.002965 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:25.003563 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:25.005014 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:25.005281 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:25.005412 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:25.011858 (+ 6446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:25.011862 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:25.011865 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:25.011896 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:25.013851 (+ 1955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:25.014601 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:25.014605 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:25.014606 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:25.014706 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:25.015196 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:25.015200 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:25.015202 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:25.015428 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:25.015555 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:25.016174 (+ 619us) server_negotiation.cc:299] Negotiation successful
0619 14:46:25.016340 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 14:46:25.017575 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:24.998549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43775 (local address 127.0.98.1:52711)
0619 14:46:24.999134 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:24.999200 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:25.000795 (+ 1595us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:25.001613 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:25.001622 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:25.002072 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:46:25.002759 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:25.002778 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:25.003720 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:25.003731 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:25.004858 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:25.004869 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:25.005489 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:25.005494 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:25.005690 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:25.006451 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:46:25.006471 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:46:25.011587 (+ 5116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:46:25.014041 (+ 2454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:25.014049 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:25.014065 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:25.014406 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:25.014821 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:25.014825 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:25.014828 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:25.015064 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:25.015556 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:46:25.015567 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:46:25.015876 (+ 309us) client_negotiation.cc:769] Sending connection context
0619 14:46:25.016115 (+ 239us) client_negotiation.cc:240] Negotiation successful
0619 14:46:25.016534 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":181,"threads_started":1}
I20260619 14:46:25.019399 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43775
I20260619 14:46:25.020012 635 heartbeater.cc:464] Registering TS with master...
I20260619 14:46:25.020778 635 heartbeater.cc:511] Master 127.0.98.62:43775 requested a full tablet report, sending...
I20260619 14:46:25.023238 444 ts_manager.cc:194] Registered new tserver with Master: 37cb7b66f3f241ec9defc52eb2dc3d43 (127.0.98.1:38119)
I20260619 14:46:25.024552 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52711
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880383132139-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880383132139-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 14:46:25.037286 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880383132139-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880383132139-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880383132139-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880383132139-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43775
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880383132139-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:46:25.175869 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:46:25.176189 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:46:25.176237 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:46:25.180531 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:46:25.180626 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:46:25.180799 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 14:46:25.187281 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880383132139-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880383132139-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880383132139-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880383132139-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43775
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880383132139-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:46:25.188457 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:46:25.189706 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:46:25.201436 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:25.201452 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:25.201817 640 server_base.cc:1034] running on GCE node
W20260619 14:46:25.201436 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:25.202428 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:46:25.203291 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:46:25.204538 640 hybrid_clock.cc:648] HybridClock initialized: now 1781880385204492 us; error 76 us; skew 500 ppm
Jun 19 14:46:25 dist-test-slave-2sb1 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880385, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:46:25.209043 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 14:46:25.210512 640 webserver.cc:460] Webserver started at http://127.0.98.2:38425/ using document root <none> and password file <none>
I20260619 14:46:25.211208 640 fs_manager.cc:359] Metadata directory not provided
I20260619 14:46:25.211303 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:46:25.211506 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:46:25.214092 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880383132139-392-0/minicluster-data/ts-1/data/instance:
uuid: "448984a7d83c4eb5bb51e64e1874a2a9"
format_stamp: "Formatted at 2026-06-19 14:46:25 on dist-test-slave-2sb1"
server_key: "209a446012a28e2303f0317ff3548611"
server_key_iv: "bd5ba09e5becb5b1d76f3ca3a3df3dd0"
server_key_version: "encryptionkey@0"
I20260619 14:46:25.214716 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880383132139-392-0/minicluster-data/ts-1/wal/instance:
uuid: "448984a7d83c4eb5bb51e64e1874a2a9"
format_stamp: "Formatted at 2026-06-19 14:46:25 on dist-test-slave-2sb1"
server_key: "209a446012a28e2303f0317ff3548611"
server_key_iv: "bd5ba09e5becb5b1d76f3ca3a3df3dd0"
server_key_version: "encryptionkey@0"
I20260619 14:46:25.219763 640 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 14:46:25.223811 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:25.225883 640 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 14:46:25.226116 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880383132139-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880383132139-392-0/minicluster-data/ts-1/wal
uuid: "448984a7d83c4eb5bb51e64e1874a2a9"
format_stamp: "Formatted at 2026-06-19 14:46:25 on dist-test-slave-2sb1"
server_key: "209a446012a28e2303f0317ff3548611"
server_key_iv: "bd5ba09e5becb5b1d76f3ca3a3df3dd0"
server_key_version: "encryptionkey@0"
I20260619 14:46:25.226238 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880383132139-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880383132139-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880383132139-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:46:25.253906 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:46:25.258915 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:46:25.259179 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:46:25.259776 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:46:25.260774 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:46:25.260828 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:25.260870 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:46:25.260886 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:25.271981 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41525
I20260619 14:46:25.272029 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41525 every 8 connection(s)
I20260619 14:46:25.272996 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880383132139-392-0/minicluster-data/ts-1/data/info.pb
I20260619 14:46:25.277150 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 640
I20260619 14:46:25.277321 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880383132139-392-0/minicluster-data/ts-1/wal/instance
I20260619 14:46:25.277638 392 external_mini_cluster.cc:1428] Setting key 0ab06e4a3888a40929da1b55d97eac3b
Jun 19 14:46:25 dist-test-slave-2sb1 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880385, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260619 14:46:25.293458 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:25.275444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39135 (local address 127.0.98.62:43775)
0619 14:46:25.275722 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:25.275727 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:25.276735 (+ 1008us) server_negotiation.cc:407] Connection header received
0619 14:46:25.278550 (+ 1815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:25.278554 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:25.278611 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:25.278935 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:25.281091 (+ 2156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:25.281718 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:25.283624 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:25.284090 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:25.284252 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:25.287638 (+ 3386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:25.287646 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:25.287651 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:25.287691 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:25.289742 (+ 2051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:25.290733 (+ 991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:25.290737 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:25.290738 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:25.290899 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:25.291652 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:25.291655 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:25.291657 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:25.291968 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:25.292107 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:25.293001 (+ 894us) server_negotiation.cc:299] Negotiation successful
0619 14:46:25.293189 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173}
I20260619 14:46:25.294345 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:25.275872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43775 (local address 127.0.98.2:39135)
0619 14:46:25.276525 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:25.276580 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:25.278282 (+ 1702us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:25.279170 (+ 888us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:25.279180 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:25.279852 (+ 672us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:46:25.280869 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:25.280898 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:25.281880 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:25.281885 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:25.283360 (+ 1475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:25.283373 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:25.284290 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:25.284293 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:25.284531 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:25.285275 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:46:25.285293 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:46:25.287377 (+ 2084us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:46:25.290098 (+ 2721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:25.290106 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:25.290123 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:25.290453 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:25.291169 (+ 716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:25.291176 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:25.291181 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:25.291400 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:25.292156 (+ 756us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:46:25.292167 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:46:25.292610 (+ 443us) client_negotiation.cc:769] Sending connection context
0619 14:46:25.292854 (+ 244us) client_negotiation.cc:240] Negotiation successful
0619 14:46:25.293257 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":127,"threads_started":1}
I20260619 14:46:25.296190 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43775
I20260619 14:46:25.296530 769 heartbeater.cc:464] Registering TS with master...
I20260619 14:46:25.297291 769 heartbeater.cc:511] Master 127.0.98.62:43775 requested a full tablet report, sending...
I20260619 14:46:25.298659 444 ts_manager.cc:194] Registered new tserver with Master: 448984a7d83c4eb5bb51e64e1874a2a9 (127.0.98.2:41525)
I20260619 14:46:25.299360 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:39135
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880383132139-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880383132139-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 14:46:25.319195 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880383132139-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880383132139-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880383132139-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880383132139-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43775
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880383132139-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:46:25.447695 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:46:25.448001 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:46:25.448052 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:46:25.452459 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:46:25.452557 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:46:25.452625 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 14:46:25.458525 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880383132139-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880383132139-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880383132139-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880383132139-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43775
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880383132139-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:46:25.459739 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:46:25.460902 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:46:25.471472 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:25.471683 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:25.471807 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:25.472348 774 server_base.cc:1034] running on GCE node
I20260619 14:46:25.473071 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:46:25.473898 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:46:25.475164 774 hybrid_clock.cc:648] HybridClock initialized: now 1781880385475093 us; error 92 us; skew 500 ppm
Jun 19 14:46:25 dist-test-slave-2sb1 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880385, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:46:25.479023 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 14:46:25.480085 774 webserver.cc:460] Webserver started at http://127.0.98.3:40083/ using document root <none> and password file <none>
I20260619 14:46:25.480762 774 fs_manager.cc:359] Metadata directory not provided
I20260619 14:46:25.480832 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:46:25.481053 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:46:25.483245 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880383132139-392-0/minicluster-data/ts-2/data/instance:
uuid: "87b0580bfefd4182be6abd87002e7f9f"
format_stamp: "Formatted at 2026-06-19 14:46:25 on dist-test-slave-2sb1"
server_key: "c48dab37111356a7e2cbc0ba1344a1fa"
server_key_iv: "2c51daa1e2f05df1a8da9dc9ac0bbc53"
server_key_version: "encryptionkey@0"
I20260619 14:46:25.483846 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880383132139-392-0/minicluster-data/ts-2/wal/instance:
uuid: "87b0580bfefd4182be6abd87002e7f9f"
format_stamp: "Formatted at 2026-06-19 14:46:25 on dist-test-slave-2sb1"
server_key: "c48dab37111356a7e2cbc0ba1344a1fa"
server_key_iv: "2c51daa1e2f05df1a8da9dc9ac0bbc53"
server_key_version: "encryptionkey@0"
I20260619 14:46:25.488606 774 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260619 14:46:25.491899 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:25.493865 774 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:46:25.494073 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880383132139-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880383132139-392-0/minicluster-data/ts-2/wal
uuid: "87b0580bfefd4182be6abd87002e7f9f"
format_stamp: "Formatted at 2026-06-19 14:46:25 on dist-test-slave-2sb1"
server_key: "c48dab37111356a7e2cbc0ba1344a1fa"
server_key_iv: "2c51daa1e2f05df1a8da9dc9ac0bbc53"
server_key_version: "encryptionkey@0"
I20260619 14:46:25.494225 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880383132139-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880383132139-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880383132139-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:46:25.533730 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:46:25.538233 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:46:25.538473 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:46:25.539119 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:46:25.540011 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:46:25.540087 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:25.540162 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:46:25.540216 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:25.550734 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37849
I20260619 14:46:25.550751 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37849 every 8 connection(s)
I20260619 14:46:25.551766 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880383132139-392-0/minicluster-data/ts-2/data/info.pb
I20260619 14:46:25.558317 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 774
I20260619 14:46:25.558485 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781880383132139-392-0/minicluster-data/ts-2/wal/instance
I20260619 14:46:25.558861 392 external_mini_cluster.cc:1428] Setting key eea7811d3b397c8dc8e1ea90396e8bd0
Jun 19 14:46:25 dist-test-slave-2sb1 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880385, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:46:25.569729 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:25.553945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60481 (local address 127.0.98.62:43775)
0619 14:46:25.554159 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:25.554164 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:25.555094 (+ 930us) server_negotiation.cc:407] Connection header received
0619 14:46:25.556253 (+ 1159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:25.556256 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:25.556307 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:25.556407 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:25.557937 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:25.558553 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:25.560387 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:25.560821 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:25.560967 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:25.564291 (+ 3324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:25.564294 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:25.564296 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:25.564322 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:25.566686 (+ 2364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:25.567533 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:25.567562 (+ 29us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:25.567566 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:25.567675 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:25.568138 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:25.568141 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:25.568143 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:25.568351 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:25.568444 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:25.569331 (+ 887us) server_negotiation.cc:299] Negotiation successful
0619 14:46:25.569459 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260619 14:46:25.570636 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:25.554360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43775 (local address 127.0.98.3:60481)
0619 14:46:25.554902 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:25.554959 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:25.555861 (+ 902us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:25.556762 (+ 901us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:25.556771 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:25.557192 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:46:25.557752 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:25.557768 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:25.558729 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:25.558735 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:25.560230 (+ 1495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:25.560251 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:25.561185 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:25.561191 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:25.561448 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:25.562268 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:46:25.562286 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:46:25.564072 (+ 1786us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:46:25.566861 (+ 2789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:25.566869 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:25.566886 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:25.567396 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:25.567796 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:25.567800 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:25.567801 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:25.567973 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:25.568475 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:46:25.568490 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:46:25.568905 (+ 415us) client_negotiation.cc:769] Sending connection context
0619 14:46:25.569258 (+ 353us) client_negotiation.cc:240] Negotiation successful
0619 14:46:25.569585 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":144,"threads_started":1}
I20260619 14:46:25.572487 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43775
I20260619 14:46:25.572950 903 heartbeater.cc:464] Registering TS with master...
I20260619 14:46:25.573750 903 heartbeater.cc:511] Master 127.0.98.62:43775 requested a full tablet report, sending...
I20260619 14:46:25.575196 444 ts_manager.cc:194] Registered new tserver with Master: 87b0580bfefd4182be6abd87002e7f9f (127.0.98.3:37849)
I20260619 14:46:25.575850 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:60481
I20260619 14:46:25.575999 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 14:46:25 dist-test-slave-2sb1 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880385, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 14:46:25 dist-test-slave-2sb1 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880385, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 14:46:25.609910 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:25.596809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38918 (local address 127.0.98.1:38119)
0619 14:46:25.597186 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:25.597195 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:25.597223 (+ 28us) server_negotiation.cc:407] Connection header received
0619 14:46:25.597329 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:25.597335 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:25.597555 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:25.597703 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:25.598601 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:25.599536 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:25.600523 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:25.601067 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:25.601180 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:25.604326 (+ 3146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:25.604338 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:25.604354 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:25.604386 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:25.606796 (+ 2410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:25.607524 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:25.607532 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:25.607533 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:25.607663 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:25.608381 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:25.608385 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:25.608386 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:25.608892 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:25.609092 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:25.609596 (+ 504us) server_negotiation.cc:299] Negotiation successful
0619 14:46:25.609759 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":141,"threads_started":1}
W20260619 14:46:25.611349 548 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38918
Jun 19 14:46:25 dist-test-slave-2sb1 krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880385, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 14:46:25 dist-test-slave-2sb1 krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880385, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 14:46:25.644290 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:25.633381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38932 (local address 127.0.98.1:38119)
0619 14:46:25.633593 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:25.633598 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:25.633864 (+ 266us) server_negotiation.cc:407] Connection header received
0619 14:46:25.634177 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:25.634182 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:25.634245 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:25.634358 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:25.635195 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:25.635743 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:25.636622 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:25.636895 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:25.637015 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:25.640214 (+ 3199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:25.640218 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:25.640220 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:25.640252 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:25.642096 (+ 1844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:25.642649 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:25.642654 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:25.642656 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:25.642749 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:25.643180 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:25.643182 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:25.643184 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:25.643393 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:25.643530 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:25.643953 (+ 423us) server_negotiation.cc:299] Negotiation successful
0619 14:46:25.644105 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260619 14:46:25.645882 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 506
I20260619 14:46:25.654043 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 640
I20260619 14:46:25.666190 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 774
I20260619 14:46:25.673868 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 412
2026-06-19T14:46:25Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2529 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880383132139-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:46:25 dist-test-slave-2sb1 krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:46:25 dist-test-slave-2sb1 krb5kdc[917](info): set up 2 sockets
Jun 19 14:46:25 dist-test-slave-2sb1 krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260619 14:46:26.784624 392 mini_kdc.cc:121] Time spent starting KDC: real 1.086s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880383132139-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880383132139-392-0/krb5kdc/test-user.keytab.
Jun 19 14:46:26 dist-test-slave-2sb1 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880386, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:46:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:46:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 14:46:26.925631 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880383132139-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880383132139-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880383132139-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38655
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38655
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:46:27.046082 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:46:27.046460 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:46:27.046541 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:46:27.051818 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:46:27.051942 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:46:27.051990 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:46:27.052055 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:46:27.052102 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:46:27.062614 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880383132139-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880383132139-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38655
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38655
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880383132139-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:46:27.064193 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:46:27.065461 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:46:27.076297 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:27.076417 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:27.076980 930 server_base.cc:1034] running on GCE node
W20260619 14:46:27.077034 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:27.077941 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:46:27.079244 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:46:27.080433 930 hybrid_clock.cc:648] HybridClock initialized: now 1781880387080392 us; error 62 us; skew 500 ppm
Jun 19 14:46:27 dist-test-slave-2sb1 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880387, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:46:27.083861 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 14:46:27.085037 930 webserver.cc:460] Webserver started at http://127.0.98.62:34151/ using document root <none> and password file <none>
I20260619 14:46:27.085844 930 fs_manager.cc:359] Metadata directory not provided
I20260619 14:46:27.085951 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:46:27.086210 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:46:27.088515 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880383132139-392-0/minicluster-data/master-0/data/instance:
uuid: "907a04895c12476f9062a6e1f234da79"
format_stamp: "Formatted at 2026-06-19 14:46:27 on dist-test-slave-2sb1"
server_key: "97ba7e0910f45dab4c1d921a77ba9da8"
server_key_iv: "2f85ce1a372df44c2a163d71b2a138ef"
server_key_version: "encryptionkey@0"
I20260619 14:46:27.089226 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880383132139-392-0/minicluster-data/master-0/wal/instance:
uuid: "907a04895c12476f9062a6e1f234da79"
format_stamp: "Formatted at 2026-06-19 14:46:27 on dist-test-slave-2sb1"
server_key: "97ba7e0910f45dab4c1d921a77ba9da8"
server_key_iv: "2f85ce1a372df44c2a163d71b2a138ef"
server_key_version: "encryptionkey@0"
I20260619 14:46:27.094496 930 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 14:46:27.098369 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:27.100301 930 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260619 14:46:27.100484 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880383132139-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880383132139-392-0/minicluster-data/master-0/wal
uuid: "907a04895c12476f9062a6e1f234da79"
format_stamp: "Formatted at 2026-06-19 14:46:27 on dist-test-slave-2sb1"
server_key: "97ba7e0910f45dab4c1d921a77ba9da8"
server_key_iv: "2f85ce1a372df44c2a163d71b2a138ef"
server_key_version: "encryptionkey@0"
I20260619 14:46:27.100636 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880383132139-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880383132139-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880383132139-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:46:27.111925 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:46:27.115857 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:46:27.116114 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:46:27.124784 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38655
I20260619 14:46:27.124907 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38655 every 8 connection(s)
I20260619 14:46:27.125878 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:46:27.129108 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:46:27.133687 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 930
I20260619 14:46:27.133831 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880383132139-392-0/minicluster-data/master-0/wal/instance
I20260619 14:46:27.134097 392 external_mini_cluster.cc:1428] Setting key bd9054233ade77816637b8305d90b782
I20260619 14:46:27.135787 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 907a04895c12476f9062a6e1f234da79: Bootstrap starting.
I20260619 14:46:27.138309 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 907a04895c12476f9062a6e1f234da79: Neither blocks nor log segments found. Creating new log.
I20260619 14:46:27.139137 998 log.cc:824] T 00000000000000000000000000000000 P 907a04895c12476f9062a6e1f234da79: Log is configured to *not* fsync() on all Append() calls
I20260619 14:46:27.141314 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 907a04895c12476f9062a6e1f234da79: No bootstrap required, opened a new log
Jun 19 14:46:27 dist-test-slave-2sb1 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880386, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:46:27.145797 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 907a04895c12476f9062a6e1f234da79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "907a04895c12476f9062a6e1f234da79" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38655 } }
I20260619 14:46:27.146139 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 907a04895c12476f9062a6e1f234da79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:46:27.146210 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 907a04895c12476f9062a6e1f234da79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 907a04895c12476f9062a6e1f234da79, State: Initialized, Role: FOLLOWER
I20260619 14:46:27.146804 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 907a04895c12476f9062a6e1f234da79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "907a04895c12476f9062a6e1f234da79" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38655 } }
I20260619 14:46:27.146986 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 907a04895c12476f9062a6e1f234da79 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:46:27.147051 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 907a04895c12476f9062a6e1f234da79 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:46:27.147186 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 907a04895c12476f9062a6e1f234da79 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:46:27.148223 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 907a04895c12476f9062a6e1f234da79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "907a04895c12476f9062a6e1f234da79" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38655 } }
I20260619 14:46:27.148646 998 leader_election.cc:302] T 00000000000000000000000000000000 P 907a04895c12476f9062a6e1f234da79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 907a04895c12476f9062a6e1f234da79; no voters:
I20260619 14:46:27.149087 998 leader_election.cc:288] T 00000000000000000000000000000000 P 907a04895c12476f9062a6e1f234da79 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:46:27.149456 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 907a04895c12476f9062a6e1f234da79 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:46:27.149823 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P 907a04895c12476f9062a6e1f234da79 [term 1 LEADER]: Becoming Leader. State: Replica: 907a04895c12476f9062a6e1f234da79, State: Running, Role: LEADER
I20260619 14:46:27.150519 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P 907a04895c12476f9062a6e1f234da79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "907a04895c12476f9062a6e1f234da79" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38655 } }
I20260619 14:46:27.150877 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 907a04895c12476f9062a6e1f234da79 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:46:27.152293 1003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 907a04895c12476f9062a6e1f234da79 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 907a04895c12476f9062a6e1f234da79. Latest consensus state: current_term: 1 leader_uuid: "907a04895c12476f9062a6e1f234da79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "907a04895c12476f9062a6e1f234da79" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38655 } } }
I20260619 14:46:27.152432 1003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 907a04895c12476f9062a6e1f234da79 [sys.catalog]: This master's current role is: LEADER
I20260619 14:46:27.153069 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 907a04895c12476f9062a6e1f234da79 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "907a04895c12476f9062a6e1f234da79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "907a04895c12476f9062a6e1f234da79" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38655 } } }
I20260619 14:46:27.153235 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 907a04895c12476f9062a6e1f234da79 [sys.catalog]: This master's current role is: LEADER
I20260619 14:46:27.153467 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:27.135852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59058 (local address 127.0.98.62:38655)
0619 14:46:27.136339 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:27.136358 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:27.136395 (+ 37us) server_negotiation.cc:407] Connection header received
0619 14:46:27.137158 (+ 763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:27.137179 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:27.137548 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:27.137922 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:27.138784 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:27.139882 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:27.140871 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:27.141369 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:27.141549 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:27.144494 (+ 2945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:27.144502 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:27.144524 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:27.144561 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:27.147370 (+ 2809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:27.148433 (+ 1063us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:27.148442 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:27.148449 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:27.148619 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:27.149815 (+ 1196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:27.149820 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:27.149822 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:27.150815 (+ 993us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:27.151041 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:27.151512 (+ 471us) server_negotiation.cc:299] Negotiation successful
0619 14:46:27.152066 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":154,"threads_started":1}
I20260619 14:46:27.154533 1012 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:46:27.158790 1012 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:46:27.166343 1012 catalog_manager.cc:1347] Generated new cluster ID: 3387c2e701bf4072b5541e85523cf87e
I20260619 14:46:27.166512 1012 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:46:27.188200 1012 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:46:27.189273 1012 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:46:27.201339 1012 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 907a04895c12476f9062a6e1f234da79: Generated new TSK 0
I20260619 14:46:27.202510 1012 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880383132139-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880383132139-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 14:46:27.249058 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880383132139-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880383132139-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880383132139-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880383132139-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38655
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880383132139-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:46:27.378510 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:46:27.378851 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:46:27.378901 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:46:27.384130 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:46:27.384245 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:46:27.384320 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 14:46:27.390676 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880383132139-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880383132139-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880383132139-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880383132139-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38655
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880383132139-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:46:27.391855 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:46:27.393014 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:46:27.403981 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:27.404150 1024 server_base.cc:1034] running on GCE node
W20260619 14:46:27.404172 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:27.404117 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:27.404965 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:46:27.405738 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:46:27.406939 1024 hybrid_clock.cc:648] HybridClock initialized: now 1781880387406920 us; error 38 us; skew 500 ppm
Jun 19 14:46:27 dist-test-slave-2sb1 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880387, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:46:27.410521 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 14:46:27.411615 1024 webserver.cc:460] Webserver started at http://127.0.98.1:46735/ using document root <none> and password file <none>
I20260619 14:46:27.412245 1024 fs_manager.cc:359] Metadata directory not provided
I20260619 14:46:27.412309 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:46:27.412530 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:46:27.414717 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880383132139-392-0/minicluster-data/ts-0/data/instance:
uuid: "d74ea25750374f53becc3049afab43ee"
format_stamp: "Formatted at 2026-06-19 14:46:27 on dist-test-slave-2sb1"
server_key: "537e39a9dbecddcce7811f878d2fbfcf"
server_key_iv: "a059f774688f033c8427ae7d1fc7463a"
server_key_version: "encryptionkey@0"
I20260619 14:46:27.415303 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880383132139-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d74ea25750374f53becc3049afab43ee"
format_stamp: "Formatted at 2026-06-19 14:46:27 on dist-test-slave-2sb1"
server_key: "537e39a9dbecddcce7811f878d2fbfcf"
server_key_iv: "a059f774688f033c8427ae7d1fc7463a"
server_key_version: "encryptionkey@0"
I20260619 14:46:27.419431 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 14:46:27.422497 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:27.424103 1024 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 14:46:27.424243 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880383132139-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880383132139-392-0/minicluster-data/ts-0/wal
uuid: "d74ea25750374f53becc3049afab43ee"
format_stamp: "Formatted at 2026-06-19 14:46:27 on dist-test-slave-2sb1"
server_key: "537e39a9dbecddcce7811f878d2fbfcf"
server_key_iv: "a059f774688f033c8427ae7d1fc7463a"
server_key_version: "encryptionkey@0"
I20260619 14:46:27.424422 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880383132139-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880383132139-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880383132139-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:46:27.449649 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:46:27.453939 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:46:27.454171 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:46:27.454808 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:46:27.455681 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:46:27.455732 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:27.455773 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:46:27.455828 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:27.465065 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39197
I20260619 14:46:27.465112 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39197 every 8 connection(s)
I20260619 14:46:27.466035 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880383132139-392-0/minicluster-data/ts-0/data/info.pb
I20260619 14:46:27.466917 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 1024
I20260619 14:46:27.467023 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880383132139-392-0/minicluster-data/ts-0/wal/instance
I20260619 14:46:27.467381 392 external_mini_cluster.cc:1428] Setting key 79541383f1c6f7e6cdab35ada70595e5
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Jun 19 14:46:27 dist-test-slave-2sb1 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880387, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:46:27.489140 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:27.468523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45323 (local address 127.0.98.62:38655)
0619 14:46:27.468688 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:27.468693 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:27.469926 (+ 1233us) server_negotiation.cc:407] Connection header received
0619 14:46:27.472270 (+ 2344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:27.472275 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:27.472343 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:27.472460 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:27.474406 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:27.474977 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:27.477138 (+ 2161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:27.477394 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:27.477504 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:27.484208 (+ 6704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:27.484212 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:27.484215 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:27.484247 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:27.486398 (+ 2151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:27.487246 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:27.487249 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:27.487250 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:27.487336 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:27.487838 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:27.487841 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:27.487842 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:27.488061 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:27.488195 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:27.488827 (+ 632us) server_negotiation.cc:299] Negotiation successful
0619 14:46:27.488969 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 14:46:27.490065 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:27.469128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38655 (local address 127.0.98.1:45323)
0619 14:46:27.469731 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:27.469799 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:27.471848 (+ 2049us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:27.472807 (+ 959us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:27.472821 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:27.473412 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:46:27.474207 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:27.474235 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:27.475137 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:27.475144 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:27.476944 (+ 1800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:27.476961 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:27.477669 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:27.477681 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:27.477966 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:27.478809 (+ 843us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:46:27.478832 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:46:27.483922 (+ 5090us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:46:27.486581 (+ 2659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:27.486593 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:27.486617 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:27.487091 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:27.487448 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:27.487455 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:27.487460 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:27.487710 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:27.488195 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:46:27.488204 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:46:27.488495 (+ 291us) client_negotiation.cc:769] Sending connection context
0619 14:46:27.488762 (+ 267us) client_negotiation.cc:240] Negotiation successful
0619 14:46:27.489143 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":129,"threads_started":1}
I20260619 14:46:27.491778 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38655
I20260619 14:46:27.492095 1153 heartbeater.cc:464] Registering TS with master...
I20260619 14:46:27.492848 1153 heartbeater.cc:511] Master 127.0.98.62:38655 requested a full tablet report, sending...
I20260619 14:46:27.494977 959 ts_manager.cc:194] Registered new tserver with Master: d74ea25750374f53becc3049afab43ee (127.0.98.1:39197)
I20260619 14:46:27.496598 959 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45323
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880383132139-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880383132139-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 14:46:27.505849 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880383132139-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880383132139-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880383132139-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880383132139-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38655
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880383132139-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:46:27.635398 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:46:27.635722 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:46:27.635771 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:46:27.640482 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:46:27.640581 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:46:27.640755 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 14:46:27.646970 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880383132139-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880383132139-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880383132139-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880383132139-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38655
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880383132139-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:46:27.648221 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:46:27.649351 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:46:27.660502 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:27.660743 1158 server_base.cc:1034] running on GCE node
W20260619 14:46:27.660825 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:27.660506 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:27.661330 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:46:27.662063 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:46:27.663285 1158 hybrid_clock.cc:648] HybridClock initialized: now 1781880387663268 us; error 33 us; skew 500 ppm
Jun 19 14:46:27 dist-test-slave-2sb1 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880387, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:46:27.666955 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 14:46:27.668036 1158 webserver.cc:460] Webserver started at http://127.0.98.2:45163/ using document root <none> and password file <none>
I20260619 14:46:27.668821 1158 fs_manager.cc:359] Metadata directory not provided
I20260619 14:46:27.668922 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:46:27.669178 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:46:27.671422 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880383132139-392-0/minicluster-data/ts-1/data/instance:
uuid: "5354341d2e0a4d8183e7606bf4c107ba"
format_stamp: "Formatted at 2026-06-19 14:46:27 on dist-test-slave-2sb1"
server_key: "c6626f1f7b7649ae992536de3210b8e9"
server_key_iv: "60e3702dec78aa4c0082fee0ad78b7d3"
server_key_version: "encryptionkey@0"
I20260619 14:46:27.672084 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880383132139-392-0/minicluster-data/ts-1/wal/instance:
uuid: "5354341d2e0a4d8183e7606bf4c107ba"
format_stamp: "Formatted at 2026-06-19 14:46:27 on dist-test-slave-2sb1"
server_key: "c6626f1f7b7649ae992536de3210b8e9"
server_key_iv: "60e3702dec78aa4c0082fee0ad78b7d3"
server_key_version: "encryptionkey@0"
I20260619 14:46:27.676748 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20260619 14:46:27.679683 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:27.681169 1158 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:46:27.681339 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880383132139-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880383132139-392-0/minicluster-data/ts-1/wal
uuid: "5354341d2e0a4d8183e7606bf4c107ba"
format_stamp: "Formatted at 2026-06-19 14:46:27 on dist-test-slave-2sb1"
server_key: "c6626f1f7b7649ae992536de3210b8e9"
server_key_iv: "60e3702dec78aa4c0082fee0ad78b7d3"
server_key_version: "encryptionkey@0"
I20260619 14:46:27.681485 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880383132139-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880383132139-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880383132139-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:46:27.695178 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:46:27.699399 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:46:27.699666 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:46:27.700320 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:46:27.701318 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:46:27.701395 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:27.701464 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:46:27.701498 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:27.711439 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41369
I20260619 14:46:27.711516 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41369 every 8 connection(s)
I20260619 14:46:27.712481 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880383132139-392-0/minicluster-data/ts-1/data/info.pb
I20260619 14:46:27.713937 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 1158
I20260619 14:46:27.714061 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880383132139-392-0/minicluster-data/ts-1/wal/instance
I20260619 14:46:27.714325 392 external_mini_cluster.cc:1428] Setting key ec484535515c6384b30f1cf4183a92c3
Jun 19 14:46:27 dist-test-slave-2sb1 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880387, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260619 14:46:27.729126 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:27.714751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50551 (local address 127.0.98.62:38655)
0619 14:46:27.714906 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:27.714910 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:27.715848 (+ 938us) server_negotiation.cc:407] Connection header received
0619 14:46:27.717329 (+ 1481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:27.717332 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:27.717384 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:27.717461 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:27.718671 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:27.719200 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:27.720412 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:27.720709 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:27.720817 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:27.723818 (+ 3001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:27.723822 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:27.723824 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:27.723849 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:27.725811 (+ 1962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:27.726557 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:27.726560 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:27.726562 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:27.726645 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:27.727209 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:27.727215 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:27.727218 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:27.727620 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:27.727764 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:27.728617 (+ 853us) server_negotiation.cc:299] Negotiation successful
0619 14:46:27.728821 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 14:46:27.730149 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:27.715189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38655 (local address 127.0.98.2:50551)
0619 14:46:27.715681 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:27.715735 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:27.717108 (+ 1373us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:27.717641 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:27.717648 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:27.718033 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:46:27.718526 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:27.718541 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:27.719336 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:27.719345 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:27.720266 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:27.720277 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:27.720878 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:27.720882 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:27.721061 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:27.721874 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:46:27.721892 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:46:27.723565 (+ 1673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:46:27.725957 (+ 2392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:27.725963 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:27.725978 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:27.726432 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:27.726786 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:27.726789 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:27.726790 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:27.727046 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:27.727801 (+ 755us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:46:27.727809 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:46:27.728215 (+ 406us) client_negotiation.cc:769] Sending connection context
0619 14:46:27.728574 (+ 359us) client_negotiation.cc:240] Negotiation successful
0619 14:46:27.729127 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":142,"threads_started":1}
I20260619 14:46:27.731472 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38655
I20260619 14:46:27.731797 1287 heartbeater.cc:464] Registering TS with master...
I20260619 14:46:27.732482 1287 heartbeater.cc:511] Master 127.0.98.62:38655 requested a full tablet report, sending...
I20260619 14:46:27.733783 959 ts_manager.cc:194] Registered new tserver with Master: 5354341d2e0a4d8183e7606bf4c107ba (127.0.98.2:41369)
I20260619 14:46:27.734386 959 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50551
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880383132139-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880383132139-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 14:46:27.750537 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880383132139-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880383132139-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880383132139-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880383132139-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38655
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880383132139-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:46:27.879555 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:46:27.879892 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:46:27.879944 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:46:27.884348 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:46:27.884440 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:46:27.884509 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 14:46:27.891258 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880383132139-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880383132139-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880383132139-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880383132139-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38655
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880383132139-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:46:27.892868 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:46:27.894183 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:46:27.906502 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:27.906491 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:27.906518 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:27.906947 1292 server_base.cc:1034] running on GCE node
I20260619 14:46:27.907534 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:46:27.908234 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:46:27.909435 1292 hybrid_clock.cc:648] HybridClock initialized: now 1781880387909419 us; error 37 us; skew 500 ppm
Jun 19 14:46:27 dist-test-slave-2sb1 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880387, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:46:27.912572 1292 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 14:46:27.913682 1292 webserver.cc:460] Webserver started at http://127.0.98.3:46727/ using document root <none> and password file <none>
I20260619 14:46:27.914287 1292 fs_manager.cc:359] Metadata directory not provided
I20260619 14:46:27.914356 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:46:27.914557 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:46:27.916958 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880383132139-392-0/minicluster-data/ts-2/data/instance:
uuid: "5340184475c346d687353205eac8e616"
format_stamp: "Formatted at 2026-06-19 14:46:27 on dist-test-slave-2sb1"
server_key: "92056025ad6a06674259a1c0d52676b7"
server_key_iv: "f6d3bd7f85ea1807712acaabccb460ca"
server_key_version: "encryptionkey@0"
I20260619 14:46:27.917603 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880383132139-392-0/minicluster-data/ts-2/wal/instance:
uuid: "5340184475c346d687353205eac8e616"
format_stamp: "Formatted at 2026-06-19 14:46:27 on dist-test-slave-2sb1"
server_key: "92056025ad6a06674259a1c0d52676b7"
server_key_iv: "f6d3bd7f85ea1807712acaabccb460ca"
server_key_version: "encryptionkey@0"
I20260619 14:46:27.922438 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260619 14:46:27.925812 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:27.927505 1292 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:46:27.927645 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880383132139-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880383132139-392-0/minicluster-data/ts-2/wal
uuid: "5340184475c346d687353205eac8e616"
format_stamp: "Formatted at 2026-06-19 14:46:27 on dist-test-slave-2sb1"
server_key: "92056025ad6a06674259a1c0d52676b7"
server_key_iv: "f6d3bd7f85ea1807712acaabccb460ca"
server_key_version: "encryptionkey@0"
I20260619 14:46:27.927758 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880383132139-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880383132139-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880383132139-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:46:27.943770 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:46:27.948112 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:46:27.948303 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:46:27.949090 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:46:27.950052 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:46:27.950136 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:27.950227 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:46:27.950286 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:27.961375 1292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35079
I20260619 14:46:27.961397 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35079 every 8 connection(s)
I20260619 14:46:27.962535 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880383132139-392-0/minicluster-data/ts-2/data/info.pb
I20260619 14:46:27.969506 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 1292
I20260619 14:46:27.969676 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781880383132139-392-0/minicluster-data/ts-2/wal/instance
I20260619 14:46:27.970002 392 external_mini_cluster.cc:1428] Setting key b82f4a0f87402c4d68738beaff0c5c9d
Jun 19 14:46:27 dist-test-slave-2sb1 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880387, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:46:27.982088 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:27.965138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55309 (local address 127.0.98.62:38655)
0619 14:46:27.965315 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:27.965321 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:27.966352 (+ 1031us) server_negotiation.cc:407] Connection header received
0619 14:46:27.967956 (+ 1604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:27.967960 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:27.968015 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:27.968110 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:27.969944 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:27.970684 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:27.972408 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:27.972884 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:27.973192 (+ 308us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:27.976505 (+ 3313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:27.976508 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:27.976510 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:27.976539 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:27.978635 (+ 2096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:27.979640 (+ 1005us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:27.979653 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:27.979655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:27.979754 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:27.980365 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:27.980368 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:27.980369 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:27.980771 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:27.980917 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:27.981731 (+ 814us) server_negotiation.cc:299] Negotiation successful
0619 14:46:27.981892 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 14:46:27.983098 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:27.965529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38655 (local address 127.0.98.3:55309)
0619 14:46:27.966097 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:27.966173 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:27.967685 (+ 1512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:27.968441 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:27.968452 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:27.969067 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:46:27.969684 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:27.969701 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:27.970897 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:27.970901 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:27.972275 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:27.972287 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:27.973182 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:27.973191 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:27.973585 (+ 394us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:27.974412 (+ 827us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:46:27.974431 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:46:27.976224 (+ 1793us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:46:27.978858 (+ 2634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:27.978865 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:27.978908 (+ 43us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:27.979418 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:27.979912 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:27.979917 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:27.979919 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:27.980210 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:27.980928 (+ 718us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:46:27.980936 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:46:27.981357 (+ 421us) client_negotiation.cc:769] Sending connection context
0619 14:46:27.981641 (+ 284us) client_negotiation.cc:240] Negotiation successful
0619 14:46:27.982041 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":146,"threads_started":1}
I20260619 14:46:27.984951 1421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38655
I20260619 14:46:27.985392 1421 heartbeater.cc:464] Registering TS with master...
I20260619 14:46:27.986023 1421 heartbeater.cc:511] Master 127.0.98.62:38655 requested a full tablet report, sending...
I20260619 14:46:27.987288 959 ts_manager.cc:194] Registered new tserver with Master: 5340184475c346d687353205eac8e616 (127.0.98.3:35079)
I20260619 14:46:27.987887 959 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:55309
I20260619 14:46:27.998363 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:46:28.012048 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:28.002509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59068 (local address 127.0.98.62:38655)
0619 14:46:28.002689 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:28.002694 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:28.002970 (+ 276us) server_negotiation.cc:407] Connection header received
0619 14:46:28.003271 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:28.003274 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:28.003330 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:28.003420 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:28.004328 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:28.004986 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:28.005862 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:28.006114 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:28.006232 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:28.007871 (+ 1639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:28.007875 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:28.007877 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:28.007905 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:28.010010 (+ 2105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:28.010584 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:28.010587 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:28.010588 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:28.010672 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:28.011011 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:28.011014 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:28.011016 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:28.011268 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:28.011381 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:28.011705 (+ 324us) server_negotiation.cc:299] Negotiation successful
0619 14:46:28.011819 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260619 14:46:28.017668 959 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59068:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:46:28.021968 959 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:46:28.039811 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:28.032518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41369 (local address 127.0.0.1:51476)
0619 14:46:28.033126 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:28.033181 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:28.033327 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:28.034390 (+ 1063us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:28.034396 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:28.034417 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:46:28.034591 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:28.034598 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:28.036164 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:28.036168 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:28.038057 (+ 1889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:28.038068 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:28.039230 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:28.039234 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:28.039358 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:28.039393 (+ 35us) client_negotiation.cc:769] Sending connection context
0619 14:46:28.039482 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 14:46:28.039638 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"mutex_wait_us":186,"spinlock_wait_cycles":12544,"thread_start_us":266,"threads_started":1}
I20260619 14:46:28.039882 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:28.033500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37320 (local address 127.0.98.1:39197)
0619 14:46:28.034890 (+ 1390us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:28.034895 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:28.034915 (+ 20us) server_negotiation.cc:407] Connection header received
0619 14:46:28.034977 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:28.034980 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:28.035132 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:28.035351 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:46:28.035605 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:28.036622 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:28.038068 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:28.038942 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:28.039079 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:28.039264 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:28.039638 (+ 374us) server_negotiation.cc:299] Negotiation successful
0619 14:46:28.039743 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1274,"thread_start_us":98,"threads_started":1}
I20260619 14:46:28.040195 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:28.032702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39197 (local address 127.0.0.1:37320)
0619 14:46:28.033126 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:28.033181 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:28.033327 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:28.035333 (+ 2006us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:28.035336 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:28.035356 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:46:28.035458 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:28.035468 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:28.036810 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:28.036817 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:28.037932 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:28.037941 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:28.039125 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:28.039132 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:28.039419 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:28.039481 (+ 62us) client_negotiation.cc:769] Sending connection context
0619 14:46:28.039579 (+ 98us) client_negotiation.cc:240] Negotiation successful
0619 14:46:28.039735 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"mutex_wait_us":220,"spinlock_wait_cycles":768,"thread_start_us":133,"threads_started":1}
I20260619 14:46:28.040565 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:28.033715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45738 (local address 127.0.98.3:35079)
0619 14:46:28.034096 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:28.034106 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:28.034124 (+ 18us) server_negotiation.cc:407] Connection header received
0619 14:46:28.034182 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:28.034186 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:28.034366 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:28.034506 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:46:28.034838 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:28.035993 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:28.038373 (+ 2380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:28.039007 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:28.039164 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:28.039324 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:28.039908 (+ 584us) server_negotiation.cc:299] Negotiation successful
0619 14:46:28.040355 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":84,"threads_started":1}
I20260619 14:46:28.040721 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:28.033580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35079 (local address 127.0.0.1:45738)
0619 14:46:28.033866 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:28.033881 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:28.033970 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:28.034598 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:28.034602 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:28.034619 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:46:28.034724 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:28.034729 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:28.036845 (+ 2116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:28.036848 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:28.038240 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:28.038247 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:28.039305 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:28.039309 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:28.039489 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:28.039508 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 14:46:28.039578 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 14:46:28.039735 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"mutex_wait_us":133,"thread_start_us":96,"threads_started":1}
I20260619 14:46:28.040798 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:28.032691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51476 (local address 127.0.98.2:41369)
0619 14:46:28.033828 (+ 1137us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:28.033842 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:28.033870 (+ 28us) server_negotiation.cc:407] Connection header received
0619 14:46:28.033958 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:28.033964 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:28.034199 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:28.034381 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:46:28.034728 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:28.036015 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:28.038211 (+ 2196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:28.039087 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:28.040297 (+ 1210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:28.040468 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:28.040522 (+ 54us) server_negotiation.cc:299] Negotiation successful
0619 14:46:28.040624 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1022,"thread_start_us":136,"threads_started":1}
I20260619 14:46:28.043895 1355 tablet_service.cc:1467] Processing CreateTablet for tablet 7ad021befa1343448c22b939dd169e1a (DEFAULT_TABLE table=test-table [id=d49f8fd01fd541fc9d38a544a00ad64b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:46:28.043898 1087 tablet_service.cc:1467] Processing CreateTablet for tablet 7ad021befa1343448c22b939dd169e1a (DEFAULT_TABLE table=test-table [id=d49f8fd01fd541fc9d38a544a00ad64b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:46:28.043905 1221 tablet_service.cc:1467] Processing CreateTablet for tablet 7ad021befa1343448c22b939dd169e1a (DEFAULT_TABLE table=test-table [id=d49f8fd01fd541fc9d38a544a00ad64b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:46:28.045068 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ad021befa1343448c22b939dd169e1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:46:28.045068 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ad021befa1343448c22b939dd169e1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:46:28.045068 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ad021befa1343448c22b939dd169e1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:46:28.052763 1438 tablet_bootstrap.cc:492] T 7ad021befa1343448c22b939dd169e1a P 5340184475c346d687353205eac8e616: Bootstrap starting.
I20260619 14:46:28.053853 1440 tablet_bootstrap.cc:492] T 7ad021befa1343448c22b939dd169e1a P 5354341d2e0a4d8183e7606bf4c107ba: Bootstrap starting.
I20260619 14:46:28.054450 1439 tablet_bootstrap.cc:492] T 7ad021befa1343448c22b939dd169e1a P d74ea25750374f53becc3049afab43ee: Bootstrap starting.
I20260619 14:46:28.055584 1438 tablet_bootstrap.cc:654] T 7ad021befa1343448c22b939dd169e1a P 5340184475c346d687353205eac8e616: Neither blocks nor log segments found. Creating new log.
I20260619 14:46:28.056560 1440 tablet_bootstrap.cc:654] T 7ad021befa1343448c22b939dd169e1a P 5354341d2e0a4d8183e7606bf4c107ba: Neither blocks nor log segments found. Creating new log.
I20260619 14:46:28.056907 1439 tablet_bootstrap.cc:654] T 7ad021befa1343448c22b939dd169e1a P d74ea25750374f53becc3049afab43ee: Neither blocks nor log segments found. Creating new log.
I20260619 14:46:28.057940 1439 log.cc:824] T 7ad021befa1343448c22b939dd169e1a P d74ea25750374f53becc3049afab43ee: Log is configured to *not* fsync() on all Append() calls
I20260619 14:46:28.058001 1440 log.cc:824] T 7ad021befa1343448c22b939dd169e1a P 5354341d2e0a4d8183e7606bf4c107ba: Log is configured to *not* fsync() on all Append() calls
I20260619 14:46:28.057989 1438 log.cc:824] T 7ad021befa1343448c22b939dd169e1a P 5340184475c346d687353205eac8e616: Log is configured to *not* fsync() on all Append() calls
I20260619 14:46:28.060256 1438 tablet_bootstrap.cc:492] T 7ad021befa1343448c22b939dd169e1a P 5340184475c346d687353205eac8e616: No bootstrap required, opened a new log
I20260619 14:46:28.060329 1439 tablet_bootstrap.cc:492] T 7ad021befa1343448c22b939dd169e1a P d74ea25750374f53becc3049afab43ee: No bootstrap required, opened a new log
I20260619 14:46:28.060329 1440 tablet_bootstrap.cc:492] T 7ad021befa1343448c22b939dd169e1a P 5354341d2e0a4d8183e7606bf4c107ba: No bootstrap required, opened a new log
I20260619 14:46:28.060532 1438 ts_tablet_manager.cc:1397] T 7ad021befa1343448c22b939dd169e1a P 5340184475c346d687353205eac8e616: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260619 14:46:28.060534 1440 ts_tablet_manager.cc:1397] T 7ad021befa1343448c22b939dd169e1a P 5354341d2e0a4d8183e7606bf4c107ba: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260619 14:46:28.060534 1439 ts_tablet_manager.cc:1397] T 7ad021befa1343448c22b939dd169e1a P d74ea25750374f53becc3049afab43ee: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260619 14:46:28.063872 1438 raft_consensus.cc:348] T 7ad021befa1343448c22b939dd169e1a P 5340184475c346d687353205eac8e616 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5354341d2e0a4d8183e7606bf4c107ba" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41369 } } peers { permanent_uuid: "5340184475c346d687353205eac8e616" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35079 } } peers { permanent_uuid: "d74ea25750374f53becc3049afab43ee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39197 } }
I20260619 14:46:28.064242 1438 raft_consensus.cc:374] T 7ad021befa1343448c22b939dd169e1a P 5340184475c346d687353205eac8e616 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:46:28.064301 1438 raft_consensus.cc:729] T 7ad021befa1343448c22b939dd169e1a P 5340184475c346d687353205eac8e616 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5340184475c346d687353205eac8e616, State: Initialized, Role: FOLLOWER
I20260619 14:46:28.064960 1438 consensus_queue.cc:260] T 7ad021befa1343448c22b939dd169e1a P 5340184475c346d687353205eac8e616 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5354341d2e0a4d8183e7606bf4c107ba" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41369 } } peers { permanent_uuid: "5340184475c346d687353205eac8e616" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35079 } } peers { permanent_uuid: "d74ea25750374f53becc3049afab43ee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39197 } }
I20260619 14:46:28.065251 1440 raft_consensus.cc:348] T 7ad021befa1343448c22b939dd169e1a P 5354341d2e0a4d8183e7606bf4c107ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5354341d2e0a4d8183e7606bf4c107ba" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41369 } } peers { permanent_uuid: "5340184475c346d687353205eac8e616" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35079 } } peers { permanent_uuid: "d74ea25750374f53becc3049afab43ee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39197 } }
I20260619 14:46:28.065598 1440 raft_consensus.cc:374] T 7ad021befa1343448c22b939dd169e1a P 5354341d2e0a4d8183e7606bf4c107ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:46:28.065716 1440 raft_consensus.cc:729] T 7ad021befa1343448c22b939dd169e1a P 5354341d2e0a4d8183e7606bf4c107ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5354341d2e0a4d8183e7606bf4c107ba, State: Initialized, Role: FOLLOWER
I20260619 14:46:28.065735 1439 raft_consensus.cc:348] T 7ad021befa1343448c22b939dd169e1a P d74ea25750374f53becc3049afab43ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5354341d2e0a4d8183e7606bf4c107ba" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41369 } } peers { permanent_uuid: "5340184475c346d687353205eac8e616" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35079 } } peers { permanent_uuid: "d74ea25750374f53becc3049afab43ee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39197 } }
I20260619 14:46:28.066044 1439 raft_consensus.cc:374] T 7ad021befa1343448c22b939dd169e1a P d74ea25750374f53becc3049afab43ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:46:28.066208 1438 ts_tablet_manager.cc:1428] T 7ad021befa1343448c22b939dd169e1a P 5340184475c346d687353205eac8e616: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 14:46:28.066210 1439 raft_consensus.cc:729] T 7ad021befa1343448c22b939dd169e1a P d74ea25750374f53becc3049afab43ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d74ea25750374f53becc3049afab43ee, State: Initialized, Role: FOLLOWER
I20260619 14:46:28.066407 1421 heartbeater.cc:503] Master 127.0.98.62:38655 was elected leader, sending a full tablet report...
I20260619 14:46:28.066990 1439 consensus_queue.cc:260] T 7ad021befa1343448c22b939dd169e1a P d74ea25750374f53becc3049afab43ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5354341d2e0a4d8183e7606bf4c107ba" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41369 } } peers { permanent_uuid: "5340184475c346d687353205eac8e616" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35079 } } peers { permanent_uuid: "d74ea25750374f53becc3049afab43ee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39197 } }
I20260619 14:46:28.067471 1440 consensus_queue.cc:260] T 7ad021befa1343448c22b939dd169e1a P 5354341d2e0a4d8183e7606bf4c107ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5354341d2e0a4d8183e7606bf4c107ba" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41369 } } peers { permanent_uuid: "5340184475c346d687353205eac8e616" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35079 } } peers { permanent_uuid: "d74ea25750374f53becc3049afab43ee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39197 } }
I20260619 14:46:28.068162 1153 heartbeater.cc:503] Master 127.0.98.62:38655 was elected leader, sending a full tablet report...
I20260619 14:46:28.068315 1439 ts_tablet_manager.cc:1428] T 7ad021befa1343448c22b939dd169e1a P d74ea25750374f53becc3049afab43ee: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260619 14:46:28.068742 1440 ts_tablet_manager.cc:1428] T 7ad021befa1343448c22b939dd169e1a P 5354341d2e0a4d8183e7606bf4c107ba: Time spent starting tablet: real 0.008s user 0.009s sys 0.000s
I20260619 14:46:28.069064 1287 heartbeater.cc:503] Master 127.0.98.62:38655 was elected leader, sending a full tablet report...
I20260619 14:46:28.071822 1446 raft_consensus.cc:482] T 7ad021befa1343448c22b939dd169e1a P 5354341d2e0a4d8183e7606bf4c107ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:46:28.072036 1446 raft_consensus.cc:504] T 7ad021befa1343448c22b939dd169e1a P 5354341d2e0a4d8183e7606bf4c107ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5354341d2e0a4d8183e7606bf4c107ba" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41369 } } peers { permanent_uuid: "5340184475c346d687353205eac8e616" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35079 } } peers { permanent_uuid: "d74ea25750374f53becc3049afab43ee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39197 } }
I20260619 14:46:28.074131 1446 leader_election.cc:288] T 7ad021befa1343448c22b939dd169e1a P 5354341d2e0a4d8183e7606bf4c107ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5340184475c346d687353205eac8e616 (127.0.98.3:35079), d74ea25750374f53becc3049afab43ee (127.0.98.1:39197)
I20260619 14:46:28.078225 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:28.074508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52399 (local address 127.0.98.3:35079)
0619 14:46:28.074666 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:28.074673 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:28.074688 (+ 15us) server_negotiation.cc:407] Connection header received
0619 14:46:28.074742 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:28.074748 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:28.074808 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:28.074884 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:46:28.075183 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:28.075742 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:28.077108 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:28.077589 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:28.077727 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:28.077797 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:28.078006 (+ 209us) server_negotiation.cc:299] Negotiation successful
0619 14:46:28.078081 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 14:46:28.078225 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:28.073751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35079 (local address 127.0.98.2:52399)
0619 14:46:28.074232 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:28.074248 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:28.074333 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:28.074892 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:28.074895 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:28.074909 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:46:28.075056 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:28.075062 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:28.075875 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:28.075879 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:28.076966 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:28.076977 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:28.077727 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:28.077731 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:28.077846 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:28.077871 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 14:46:28.077975 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 14:46:28.078075 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"mutex_wait_us":78}
I20260619 14:46:28.078358 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:28.074661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33839 (local address 127.0.98.1:39197)
0619 14:46:28.074797 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:28.074802 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:28.074816 (+ 14us) server_negotiation.cc:407] Connection header received
0619 14:46:28.074862 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:28.074865 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:28.074912 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:28.074989 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:46:28.075281 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:28.075776 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:28.077278 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:28.077732 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:28.077873 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:28.077938 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:28.078127 (+ 189us) server_negotiation.cc:299] Negotiation successful
0619 14:46:28.078204 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 14:46:28.078995 1447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:28.073848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39197 (local address 127.0.98.2:33839)
0619 14:46:28.074602 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:28.074618 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:28.074716 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:28.075025 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:28.075028 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:28.075046 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:46:28.075171 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:28.075176 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:28.075894 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:28.075899 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:28.076977 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:28.076987 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:28.077867 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:28.077871 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:28.077976 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:28.077997 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 14:46:28.078100 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 14:46:28.078174 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":661,"mutex_wait_us":124,"thread_start_us":137,"threads_started":1}
I20260619 14:46:28.079169 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ad021befa1343448c22b939dd169e1a" candidate_uuid: "5354341d2e0a4d8183e7606bf4c107ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5340184475c346d687353205eac8e616" is_pre_election: true
I20260619 14:46:28.079586 1375 raft_consensus.cc:2454] T 7ad021befa1343448c22b939dd169e1a P 5340184475c346d687353205eac8e616 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5354341d2e0a4d8183e7606bf4c107ba in term 0.
I20260619 14:46:28.079720 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ad021befa1343448c22b939dd169e1a" candidate_uuid: "5354341d2e0a4d8183e7606bf4c107ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d74ea25750374f53becc3049afab43ee" is_pre_election: true
I20260619 14:46:28.080008 1107 raft_consensus.cc:2454] T 7ad021befa1343448c22b939dd169e1a P d74ea25750374f53becc3049afab43ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5354341d2e0a4d8183e7606bf4c107ba in term 0.
I20260619 14:46:28.080147 1176 leader_election.cc:302] T 7ad021befa1343448c22b939dd169e1a P 5354341d2e0a4d8183e7606bf4c107ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5340184475c346d687353205eac8e616, 5354341d2e0a4d8183e7606bf4c107ba; no voters:
I20260619 14:46:28.080423 1446 raft_consensus.cc:2781] T 7ad021befa1343448c22b939dd169e1a P 5354341d2e0a4d8183e7606bf4c107ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:46:28.080524 1446 raft_consensus.cc:482] T 7ad021befa1343448c22b939dd169e1a P 5354341d2e0a4d8183e7606bf4c107ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:46:28.080582 1446 raft_consensus.cc:3037] T 7ad021befa1343448c22b939dd169e1a P 5354341d2e0a4d8183e7606bf4c107ba [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:46:28.081705 1446 raft_consensus.cc:504] T 7ad021befa1343448c22b939dd169e1a P 5354341d2e0a4d8183e7606bf4c107ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5354341d2e0a4d8183e7606bf4c107ba" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41369 } } peers { permanent_uuid: "5340184475c346d687353205eac8e616" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35079 } } peers { permanent_uuid: "d74ea25750374f53becc3049afab43ee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39197 } }
I20260619 14:46:28.082173 1446 leader_election.cc:288] T 7ad021befa1343448c22b939dd169e1a P 5354341d2e0a4d8183e7606bf4c107ba [CANDIDATE]: Term 1 election: Requested vote from peers 5340184475c346d687353205eac8e616 (127.0.98.3:35079), d74ea25750374f53becc3049afab43ee (127.0.98.1:39197)
I20260619 14:46:28.082746 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ad021befa1343448c22b939dd169e1a" candidate_uuid: "5354341d2e0a4d8183e7606bf4c107ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5340184475c346d687353205eac8e616"
I20260619 14:46:28.082803 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ad021befa1343448c22b939dd169e1a" candidate_uuid: "5354341d2e0a4d8183e7606bf4c107ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d74ea25750374f53becc3049afab43ee"
I20260619 14:46:28.082930 1107 raft_consensus.cc:3037] T 7ad021befa1343448c22b939dd169e1a P d74ea25750374f53becc3049afab43ee [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:46:28.082929 1375 raft_consensus.cc:3037] T 7ad021befa1343448c22b939dd169e1a P 5340184475c346d687353205eac8e616 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:46:28.084441 1375 raft_consensus.cc:2454] T 7ad021befa1343448c22b939dd169e1a P 5340184475c346d687353205eac8e616 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5354341d2e0a4d8183e7606bf4c107ba in term 1.
I20260619 14:46:28.084441 1107 raft_consensus.cc:2454] T 7ad021befa1343448c22b939dd169e1a P d74ea25750374f53becc3049afab43ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5354341d2e0a4d8183e7606bf4c107ba in term 1.
I20260619 14:46:28.084970 1177 leader_election.cc:302] T 7ad021befa1343448c22b939dd169e1a P 5354341d2e0a4d8183e7606bf4c107ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5354341d2e0a4d8183e7606bf4c107ba, d74ea25750374f53becc3049afab43ee; no voters:
I20260619 14:46:28.085448 1446 raft_consensus.cc:2781] T 7ad021befa1343448c22b939dd169e1a P 5354341d2e0a4d8183e7606bf4c107ba [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:46:28.085783 1446 raft_consensus.cc:686] T 7ad021befa1343448c22b939dd169e1a P 5354341d2e0a4d8183e7606bf4c107ba [term 1 LEADER]: Becoming Leader. State: Replica: 5354341d2e0a4d8183e7606bf4c107ba, State: Running, Role: LEADER
I20260619 14:46:28.086208 1446 consensus_queue.cc:237] T 7ad021befa1343448c22b939dd169e1a P 5354341d2e0a4d8183e7606bf4c107ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5354341d2e0a4d8183e7606bf4c107ba" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41369 } } peers { permanent_uuid: "5340184475c346d687353205eac8e616" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35079 } } peers { permanent_uuid: "d74ea25750374f53becc3049afab43ee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39197 } }
I20260619 14:46:28.090307 958 catalog_manager.cc:5495] T 7ad021befa1343448c22b939dd169e1a P 5354341d2e0a4d8183e7606bf4c107ba reported cstate change: term changed from 0 to 1, leader changed from <none> to 5354341d2e0a4d8183e7606bf4c107ba (127.0.98.2). New cstate: current_term: 1 leader_uuid: "5354341d2e0a4d8183e7606bf4c107ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5354341d2e0a4d8183e7606bf4c107ba" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41369 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5340184475c346d687353205eac8e616" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d74ea25750374f53becc3049afab43ee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39197 } health_report { overall_health: UNKNOWN } } }
Jun 19 14:46:28 dist-test-slave-2sb1 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880386, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 14:46:28.110888 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:28.100266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37332 (local address 127.0.98.1:39197)
0619 14:46:28.100432 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:28.100437 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:28.100556 (+ 119us) server_negotiation.cc:407] Connection header received
0619 14:46:28.100614 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:28.100616 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:28.100723 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:28.100797 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:28.101729 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:28.102423 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:28.103319 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:28.103555 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:28.103681 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:28.106787 (+ 3106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:28.106800 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:28.106815 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:28.106844 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:28.108769 (+ 1925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:28.109266 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:28.109274 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:28.109276 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:28.109389 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:28.109701 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:28.109704 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:28.109706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:28.109981 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:28.110368 (+ 387us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:28.110553 (+ 185us) server_negotiation.cc:299] Negotiation successful
0619 14:46:28.110690 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
Jun 19 14:46:28 dist-test-slave-2sb1 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880388, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 14:46:28 dist-test-slave-2sb1 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880388, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 14:46:28.148744 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:28.136173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37348 (local address 127.0.98.1:39197)
0619 14:46:28.136329 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:28.136334 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:28.136487 (+ 153us) server_negotiation.cc:407] Connection header received
0619 14:46:28.136601 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:28.136604 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:28.136734 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:28.136831 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:28.137815 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:28.138423 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:28.139774 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:28.140364 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:28.140482 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:28.144050 (+ 3568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:28.144054 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:28.144056 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:28.144094 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:28.146184 (+ 2090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:28.146791 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:28.146794 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:28.146795 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:28.146890 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:28.147361 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:28.147364 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:28.147365 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:28.147601 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:28.147739 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:28.148307 (+ 568us) server_negotiation.cc:299] Negotiation successful
0619 14:46:28.148500 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
Jun 19 14:46:28 dist-test-slave-2sb1 krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880388, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 14:46:28 dist-test-slave-2sb1 krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880388, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 14:46:28.182410 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:28.171485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37364 (local address 127.0.98.1:39197)
0619 14:46:28.171729 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:28.171734 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:28.171875 (+ 141us) server_negotiation.cc:407] Connection header received
0619 14:46:28.172050 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:28.172054 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:28.172105 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:28.172195 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:28.173238 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:28.173757 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:28.174647 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:28.174863 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:28.174972 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:28.178131 (+ 3159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:28.178135 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:28.178137 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:28.178168 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:28.179988 (+ 1820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:28.180554 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:28.180557 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:28.180558 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:28.180736 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:28.181256 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:28.181259 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:28.181260 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:28.181506 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:28.181636 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:28.181923 (+ 287us) server_negotiation.cc:299] Negotiation successful
0619 14:46:28.182249 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141}
W20260619 14:46:28.183957 1064 tablet_service.cc:2908] Rejecting scan request for tablet 7ad021befa1343448c22b939dd169e1a: Uninitialized: clean time has not yet been initialized
I20260619 14:46:28.186589 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 1024
I20260619 14:46:28.193398 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 1158
I20260619 14:46:28.200461 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 1292
I20260619 14:46:28.207329 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 930
2026-06-19T14:46:28Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2533 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880383132139-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:46:28 dist-test-slave-2sb1 krb5kdc[1462](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:46:28 dist-test-slave-2sb1 krb5kdc[1462](info): set up 2 sockets
Jun 19 14:46:28 dist-test-slave-2sb1 krb5kdc[1462](info): commencing operation
krb5kdc: starting...
W20260619 14:46:29.308909 392 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880383132139-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880383132139-392-0/krb5kdc/test-user.keytab.
Jun 19 14:46:29 dist-test-slave-2sb1 krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880389, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:46:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:46:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 14:46:29.461860 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880383132139-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880383132139-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880383132139-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46827
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46827
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:46:29.582898 1475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:46:29.583289 1475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:46:29.583375 1475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:46:29.587735 1475 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:46:29.587860 1475 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 14:46:29.587893 1475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:46:29.587939 1475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:46:29.587975 1475 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:46:29.588021 1475 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:46:29.593890 1475 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880383132139-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880383132139-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46827
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880383132139-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:46:29.595163 1475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:46:29.596283 1475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:46:29.605885 1480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:29.607452 1481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:29.607682 1475 server_base.cc:1034] running on GCE node
W20260619 14:46:29.607817 1483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:29.608573 1475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:46:29.609915 1475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:46:29.611241 1475 hybrid_clock.cc:648] HybridClock initialized: now 1781880389611205 us; error 52 us; skew 500 ppm
Jun 19 14:46:29 dist-test-slave-2sb1 krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880389, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:46:29.614933 1475 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 14:46:29.616189 1475 webserver.cc:460] Webserver started at http://127.0.98.62:42797/ using document root <none> and password file <none>
I20260619 14:46:29.616919 1475 fs_manager.cc:359] Metadata directory not provided
I20260619 14:46:29.616992 1475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:46:29.617272 1475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:46:29.619554 1475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880383132139-392-0/minicluster-data/master-0/data/instance:
uuid: "11ee01cd7a424422a7e03f596fd13d49"
format_stamp: "Formatted at 2026-06-19 14:46:29 on dist-test-slave-2sb1"
server_key: "90d66cabc78da6f94f5784c1bdcfce76"
server_key_iv: "5dc4427dedac7c7e2bbeca86fcc7b252"
server_key_version: "encryptionkey@0"
I20260619 14:46:29.620205 1475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880383132139-392-0/minicluster-data/master-0/wal/instance:
uuid: "11ee01cd7a424422a7e03f596fd13d49"
format_stamp: "Formatted at 2026-06-19 14:46:29 on dist-test-slave-2sb1"
server_key: "90d66cabc78da6f94f5784c1bdcfce76"
server_key_iv: "5dc4427dedac7c7e2bbeca86fcc7b252"
server_key_version: "encryptionkey@0"
I20260619 14:46:29.624584 1475 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260619 14:46:29.627720 1490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:29.629629 1475 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:46:29.629815 1475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880383132139-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880383132139-392-0/minicluster-data/master-0/wal
uuid: "11ee01cd7a424422a7e03f596fd13d49"
format_stamp: "Formatted at 2026-06-19 14:46:29 on dist-test-slave-2sb1"
server_key: "90d66cabc78da6f94f5784c1bdcfce76"
server_key_iv: "5dc4427dedac7c7e2bbeca86fcc7b252"
server_key_version: "encryptionkey@0"
I20260619 14:46:29.629966 1475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880383132139-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880383132139-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880383132139-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:46:29.654613 1475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:46:29.658295 1475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:46:29.658535 1475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:46:29.667116 1475 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46827
I20260619 14:46:29.667174 1552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46827 every 8 connection(s)
I20260619 14:46:29.668578 1475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:46:29.669551 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 1475
I20260619 14:46:29.669649 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880383132139-392-0/minicluster-data/master-0/wal/instance
I20260619 14:46:29.669941 392 external_mini_cluster.cc:1428] Setting key bafc4681eda78cd3657daeeb97e5e45c
I20260619 14:46:29.672255 1553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:46:29.678383 1553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11ee01cd7a424422a7e03f596fd13d49: Bootstrap starting.
Jun 19 14:46:29 dist-test-slave-2sb1 krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880389, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:46:29.681324 1553 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 11ee01cd7a424422a7e03f596fd13d49: Neither blocks nor log segments found. Creating new log.
I20260619 14:46:29.682245 1553 log.cc:824] T 00000000000000000000000000000000 P 11ee01cd7a424422a7e03f596fd13d49: Log is configured to *not* fsync() on all Append() calls
I20260619 14:46:29.684897 1553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11ee01cd7a424422a7e03f596fd13d49: No bootstrap required, opened a new log
I20260619 14:46:29.687007 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:29.671479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58230 (local address 127.0.98.62:46827)
0619 14:46:29.671924 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:29.671945 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:29.672168 (+ 223us) server_negotiation.cc:407] Connection header received
0619 14:46:29.673098 (+ 930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:29.673117 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:29.673454 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:29.673819 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:29.674641 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:29.675667 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:29.676565 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:29.677143 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:29.677334 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:29.680422 (+ 3088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:29.680428 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:29.680442 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:29.680474 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:29.683624 (+ 3150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:29.684208 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:29.684220 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:29.684226 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:29.684356 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:29.684742 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:29.684746 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:29.684747 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:29.685268 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:29.685482 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:29.685832 (+ 350us) server_negotiation.cc:299] Negotiation successful
0619 14:46:29.686154 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":148,"threads_started":1}
I20260619 14:46:29.688639 1553 raft_consensus.cc:348] T 00000000000000000000000000000000 P 11ee01cd7a424422a7e03f596fd13d49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11ee01cd7a424422a7e03f596fd13d49" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46827 } }
I20260619 14:46:29.688997 1553 raft_consensus.cc:374] T 00000000000000000000000000000000 P 11ee01cd7a424422a7e03f596fd13d49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:46:29.689085 1553 raft_consensus.cc:729] T 00000000000000000000000000000000 P 11ee01cd7a424422a7e03f596fd13d49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11ee01cd7a424422a7e03f596fd13d49, State: Initialized, Role: FOLLOWER
I20260619 14:46:29.689610 1553 consensus_queue.cc:260] T 00000000000000000000000000000000 P 11ee01cd7a424422a7e03f596fd13d49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11ee01cd7a424422a7e03f596fd13d49" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46827 } }
I20260619 14:46:29.689787 1553 raft_consensus.cc:388] T 00000000000000000000000000000000 P 11ee01cd7a424422a7e03f596fd13d49 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:46:29.689857 1553 raft_consensus.cc:482] T 00000000000000000000000000000000 P 11ee01cd7a424422a7e03f596fd13d49 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:46:29.689980 1553 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 11ee01cd7a424422a7e03f596fd13d49 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:46:29.691092 1553 raft_consensus.cc:504] T 00000000000000000000000000000000 P 11ee01cd7a424422a7e03f596fd13d49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11ee01cd7a424422a7e03f596fd13d49" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46827 } }
I20260619 14:46:29.691540 1553 leader_election.cc:302] T 00000000000000000000000000000000 P 11ee01cd7a424422a7e03f596fd13d49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 11ee01cd7a424422a7e03f596fd13d49; no voters:
I20260619 14:46:29.691964 1553 leader_election.cc:288] T 00000000000000000000000000000000 P 11ee01cd7a424422a7e03f596fd13d49 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:46:29.692144 1558 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 11ee01cd7a424422a7e03f596fd13d49 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:46:29.692373 1558 raft_consensus.cc:686] T 00000000000000000000000000000000 P 11ee01cd7a424422a7e03f596fd13d49 [term 1 LEADER]: Becoming Leader. State: Replica: 11ee01cd7a424422a7e03f596fd13d49, State: Running, Role: LEADER
I20260619 14:46:29.692879 1558 consensus_queue.cc:237] T 00000000000000000000000000000000 P 11ee01cd7a424422a7e03f596fd13d49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11ee01cd7a424422a7e03f596fd13d49" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46827 } }
I20260619 14:46:29.693634 1553 sys_catalog.cc:564] T 00000000000000000000000000000000 P 11ee01cd7a424422a7e03f596fd13d49 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:46:29.694943 1560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11ee01cd7a424422a7e03f596fd13d49 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 11ee01cd7a424422a7e03f596fd13d49. Latest consensus state: current_term: 1 leader_uuid: "11ee01cd7a424422a7e03f596fd13d49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11ee01cd7a424422a7e03f596fd13d49" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46827 } } }
I20260619 14:46:29.695088 1560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11ee01cd7a424422a7e03f596fd13d49 [sys.catalog]: This master's current role is: LEADER
I20260619 14:46:29.695386 1559 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11ee01cd7a424422a7e03f596fd13d49 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "11ee01cd7a424422a7e03f596fd13d49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11ee01cd7a424422a7e03f596fd13d49" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46827 } } }
I20260619 14:46:29.695601 1567 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:46:29.696902 1559 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11ee01cd7a424422a7e03f596fd13d49 [sys.catalog]: This master's current role is: LEADER
I20260619 14:46:29.699296 1567 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:46:29.706653 1567 catalog_manager.cc:1347] Generated new cluster ID: 4d59e838ad91410da724c6b72cc0d22d
I20260619 14:46:29.706758 1567 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:46:29.732287 1567 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:46:29.733788 1567 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:46:29.744383 1567 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 11ee01cd7a424422a7e03f596fd13d49: Generated new TSK 0
I20260619 14:46:29.745370 1567 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880383132139-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880383132139-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 14:46:29.781391 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880383132139-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880383132139-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880383132139-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880383132139-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46827
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880383132139-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:46:29.908120 1579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:46:29.908490 1579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:46:29.908543 1579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:46:29.912945 1579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:46:29.913039 1579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:46:29.913082 1579 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 14:46:29.913162 1579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 14:46:29.919060 1579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880383132139-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880383132139-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880383132139-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880383132139-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:46827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880383132139-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:46:29.920244 1579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:46:29.921331 1579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:46:29.932333 1584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:29.932343 1585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:29.932400 1579 server_base.cc:1034] running on GCE node
W20260619 14:46:29.932684 1587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:29.933133 1579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:46:29.933992 1579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:46:29.935211 1579 hybrid_clock.cc:648] HybridClock initialized: now 1781880389935178 us; error 48 us; skew 500 ppm
Jun 19 14:46:29 dist-test-slave-2sb1 krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880389, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:46:29.938824 1579 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 14:46:29.939981 1579 webserver.cc:460] Webserver started at http://127.0.98.1:46415/ using document root <none> and password file <none>
I20260619 14:46:29.940634 1579 fs_manager.cc:359] Metadata directory not provided
I20260619 14:46:29.940801 1579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:46:29.941064 1579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:46:29.944062 1579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880383132139-392-0/minicluster-data/ts-0/data/instance:
uuid: "5e5d18da021245748c1655fe7fed35b1"
format_stamp: "Formatted at 2026-06-19 14:46:29 on dist-test-slave-2sb1"
server_key: "a77ef8a0299d9baacce24df900350bb2"
server_key_iv: "fa214fc87ba0c73debc1901664fcf352"
server_key_version: "encryptionkey@0"
I20260619 14:46:29.944881 1579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880383132139-392-0/minicluster-data/ts-0/wal/instance:
uuid: "5e5d18da021245748c1655fe7fed35b1"
format_stamp: "Formatted at 2026-06-19 14:46:29 on dist-test-slave-2sb1"
server_key: "a77ef8a0299d9baacce24df900350bb2"
server_key_iv: "fa214fc87ba0c73debc1901664fcf352"
server_key_version: "encryptionkey@0"
I20260619 14:46:29.949342 1579 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 14:46:29.952431 1594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:29.954208 1579 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:46:29.954391 1579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880383132139-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880383132139-392-0/minicluster-data/ts-0/wal
uuid: "5e5d18da021245748c1655fe7fed35b1"
format_stamp: "Formatted at 2026-06-19 14:46:29 on dist-test-slave-2sb1"
server_key: "a77ef8a0299d9baacce24df900350bb2"
server_key_iv: "fa214fc87ba0c73debc1901664fcf352"
server_key_version: "encryptionkey@0"
I20260619 14:46:29.954538 1579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880383132139-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880383132139-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880383132139-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:46:29.999840 1579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:46:30.004801 1579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:46:30.005041 1579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:46:30.006594 1579 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:46:30.006695 1579 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:30.006768 1579 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:46:30.006822 1579 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:30.018376 1579 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46285
I20260619 14:46:30.018440 1709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46285 every 8 connection(s)
Jun 19 14:46:30 dist-test-slave-2sb1 krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880389, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:46:30.019579 1579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880383132139-392-0/minicluster-data/ts-0/data/info.pb
I20260619 14:46:30.020498 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 1579
I20260619 14:46:30.020613 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880383132139-392-0/minicluster-data/ts-0/wal/instance
I20260619 14:46:30.020954 392 external_mini_cluster.cc:1428] Setting key 8d54d28a03b7b180e6c867d32a1f2198
I20260619 14:46:30.025342 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:30.007995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41401 (local address 127.0.98.62:46827)
0619 14:46:30.008184 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:30.008189 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:30.009240 (+ 1051us) server_negotiation.cc:407] Connection header received
0619 14:46:30.010717 (+ 1477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:30.010723 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:30.010789 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:30.010872 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:30.012766 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:30.013562 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:30.014807 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:30.015143 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:30.015258 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:30.019466 (+ 4208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:30.019470 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:30.019473 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:30.019505 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:30.022469 (+ 2964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:30.023323 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:30.023330 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:30.023334 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:30.023448 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:30.023882 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:30.023888 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:30.023893 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:30.024187 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:30.024301 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:30.024970 (+ 669us) server_negotiation.cc:299] Negotiation successful
0619 14:46:30.025128 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260619 14:46:30.026346 1603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:30.008372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46827 (local address 127.0.98.1:41401)
0619 14:46:30.009045 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:30.009106 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:30.010420 (+ 1314us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:30.011117 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:30.011128 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:30.011724 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:46:30.012528 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:30.012554 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:30.013713 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:30.013717 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:30.014667 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:30.014677 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:30.015355 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:30.015361 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:30.015678 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:30.016828 (+ 1150us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:46:30.016856 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:46:30.019198 (+ 2342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:46:30.022635 (+ 3437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:30.022645 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:30.022665 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:30.023051 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:30.023565 (+ 514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:30.023569 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:30.023571 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:30.023766 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:30.024303 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:46:30.024309 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:46:30.024588 (+ 279us) client_negotiation.cc:769] Sending connection context
0619 14:46:30.024902 (+ 314us) client_negotiation.cc:240] Negotiation successful
0619 14:46:30.025446 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":197,"threads_started":1}
I20260619 14:46:30.027863 1710 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46827
I20260619 14:46:30.028033 1710 heartbeater.cc:464] Registering TS with master...
I20260619 14:46:30.028626 1710 heartbeater.cc:511] Master 127.0.98.62:46827 requested a full tablet report, sending...
I20260619 14:46:30.030598 1506 ts_manager.cc:194] Registered new tserver with Master: 5e5d18da021245748c1655fe7fed35b1 (127.0.98.1:46285)
I20260619 14:46:30.032137 1506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41401
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 14:46:30.040809 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:30.030686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50115 (local address 127.0.98.62:46827)
0619 14:46:30.030814 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:30.030819 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:30.031085 (+ 266us) server_negotiation.cc:407] Connection header received
0619 14:46:30.031209 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:30.031213 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:30.031259 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:30.031360 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:30.031957 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:30.032985 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:30.034415 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:30.034907 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:30.035136 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:30.036471 (+ 1335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:30.036474 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:30.036476 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:30.036534 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:30.038612 (+ 2078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:30.039288 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:30.039294 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:30.039296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:30.039390 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:30.039756 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:30.039759 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:30.039760 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:30.039960 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:30.040057 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:30.040400 (+ 343us) server_negotiation.cc:299] Negotiation successful
0619 14:46:30.040568 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":8,"server-negotiator.queue_time_us":50}
I20260619 14:46:30.041196 1717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:30.030594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46827 (local address 127.0.98.1:50115)
0619 14:46:30.030975 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:30.030991 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:30.031109 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:30.031399 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:30.031403 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:30.031658 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:46:30.031814 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:30.031823 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:30.033184 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:30.033190 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:30.034190 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:30.034202 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:30.035140 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:30.035144 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:30.035245 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:30.035921 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:46:30.035933 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:46:30.036285 (+ 352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:46:30.038802 (+ 2517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:30.038806 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:30.038808 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:30.039055 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:30.039522 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:30.039528 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:30.039533 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:30.039644 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:30.040070 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:46:30.040076 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:46:30.040198 (+ 122us) client_negotiation.cc:769] Sending connection context
0619 14:46:30.040326 (+ 128us) client_negotiation.cc:240] Negotiation successful
0619 14:46:30.040523 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"mutex_wait_us":1,"thread_start_us":125,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880383132139-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880383132139-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 14:46:30.062175 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880383132139-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880383132139-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880383132139-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880383132139-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46827
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880383132139-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:46:30.195887 1719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:46:30.196280 1719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:46:30.196381 1719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:46:30.201674 1719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:46:30.201756 1719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:46:30.201803 1719 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 14:46:30.201843 1719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 14:46:30.208895 1719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880383132139-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880383132139-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880383132139-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880383132139-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:46827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880383132139-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:46:30.210194 1719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:46:30.211251 1719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:46:30.222009 1724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:30.222002 1725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:30.222004 1727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:30.222544 1719 server_base.cc:1034] running on GCE node
I20260619 14:46:30.222996 1719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:46:30.223699 1719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:46:30.224828 1719 hybrid_clock.cc:648] HybridClock initialized: now 1781880390224787 us; error 58 us; skew 500 ppm
Jun 19 14:46:30 dist-test-slave-2sb1 krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880390, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:46:30.228101 1719 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 14:46:30.229517 1719 webserver.cc:460] Webserver started at http://127.0.98.2:35365/ using document root <none> and password file <none>
I20260619 14:46:30.230175 1719 fs_manager.cc:359] Metadata directory not provided
I20260619 14:46:30.230281 1719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:46:30.230541 1719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:46:30.233114 1719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880383132139-392-0/minicluster-data/ts-1/data/instance:
uuid: "e4742b727bc24e89ad40347bfc4a8942"
format_stamp: "Formatted at 2026-06-19 14:46:30 on dist-test-slave-2sb1"
server_key: "036c1e93c4017ec41a516a1763f93d7b"
server_key_iv: "932f97afa3f9d35d3b1d0e8469013952"
server_key_version: "encryptionkey@0"
I20260619 14:46:30.233774 1719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880383132139-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e4742b727bc24e89ad40347bfc4a8942"
format_stamp: "Formatted at 2026-06-19 14:46:30 on dist-test-slave-2sb1"
server_key: "036c1e93c4017ec41a516a1763f93d7b"
server_key_iv: "932f97afa3f9d35d3b1d0e8469013952"
server_key_version: "encryptionkey@0"
I20260619 14:46:30.238274 1719 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 14:46:30.241346 1734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:30.242750 1719 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:46:30.242920 1719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880383132139-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880383132139-392-0/minicluster-data/ts-1/wal
uuid: "e4742b727bc24e89ad40347bfc4a8942"
format_stamp: "Formatted at 2026-06-19 14:46:30 on dist-test-slave-2sb1"
server_key: "036c1e93c4017ec41a516a1763f93d7b"
server_key_iv: "932f97afa3f9d35d3b1d0e8469013952"
server_key_version: "encryptionkey@0"
I20260619 14:46:30.243064 1719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880383132139-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880383132139-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880383132139-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:46:30.269143 1719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:46:30.273430 1719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:46:30.273701 1719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:46:30.275388 1719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:46:30.275482 1719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:30.275552 1719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:46:30.275609 1719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 14:46:30 dist-test-slave-2sb1 krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880390, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:46:30.288291 1719 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38555
I20260619 14:46:30.288383 1849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38555 every 8 connection(s)
I20260619 14:46:30.289662 1719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880383132139-392-0/minicluster-data/ts-1/data/info.pb
I20260619 14:46:30.290439 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 1719
I20260619 14:46:30.290556 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880383132139-392-0/minicluster-data/ts-1/wal/instance
I20260619 14:46:30.290796 392 external_mini_cluster.cc:1428] Setting key 294634b9ee2b54ee307b403d49d31751
I20260619 14:46:30.292882 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:30.276727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50559 (local address 127.0.98.62:46827)
0619 14:46:30.276858 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:30.276862 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:30.278146 (+ 1284us) server_negotiation.cc:407] Connection header received
0619 14:46:30.279420 (+ 1274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:30.279425 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:30.279511 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:30.279627 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:30.281319 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:30.281990 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:30.284265 (+ 2275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:30.284755 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:30.284869 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:30.288305 (+ 3436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:30.288309 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:30.288312 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:30.288349 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:30.290027 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:30.290820 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:30.290823 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:30.290825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:30.290921 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:30.291461 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:30.291464 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:30.291465 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:30.291732 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:30.291838 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:30.292558 (+ 720us) server_negotiation.cc:299] Negotiation successful
0619 14:46:30.292752 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 14:46:30.294103 1743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:30.277353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46827 (local address 127.0.98.2:50559)
0619 14:46:30.277958 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:30.278004 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:30.279141 (+ 1137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:30.279869 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:30.279881 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:30.280466 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:46:30.281135 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:30.281152 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:30.282163 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:30.282169 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:30.284057 (+ 1888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:30.284078 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:30.284958 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:30.284963 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:30.285231 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:30.286327 (+ 1096us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:46:30.286345 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:46:30.288078 (+ 1733us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:46:30.290326 (+ 2248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:30.290331 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:30.290346 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:30.290692 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:30.291018 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:30.291025 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:30.291030 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:30.291215 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:30.291887 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:46:30.291892 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:46:30.292255 (+ 363us) client_negotiation.cc:769] Sending connection context
0619 14:46:30.292482 (+ 227us) client_negotiation.cc:240] Negotiation successful
0619 14:46:30.292907 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":165,"threads_started":1}
I20260619 14:46:30.295720 1850 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46827
I20260619 14:46:30.295869 1850 heartbeater.cc:464] Registering TS with master...
I20260619 14:46:30.296490 1850 heartbeater.cc:511] Master 127.0.98.62:46827 requested a full tablet report, sending...
I20260619 14:46:30.297817 1507 ts_manager.cc:194] Registered new tserver with Master: e4742b727bc24e89ad40347bfc4a8942 (127.0.98.2:38555)
I20260619 14:46:30.298753 1507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50559
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260619 14:46:30.306869 1857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:30.298189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46827 (local address 127.0.98.2:36257)
0619 14:46:30.298516 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:30.298528 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:30.298647 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:30.298919 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:30.298922 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:30.299189 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:46:30.299320 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:30.299327 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:30.300461 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:30.300464 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:30.301138 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:30.301149 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:30.301786 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:30.301789 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:30.301861 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:30.302423 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:46:30.302435 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:46:30.302845 (+ 410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:46:30.304834 (+ 1989us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:30.304841 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:30.304845 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:30.305081 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:30.305426 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:30.305429 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:30.305431 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:30.305518 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:30.306145 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:46:30.306148 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:46:30.306237 (+ 89us) client_negotiation.cc:769] Sending connection context
0619 14:46:30.306376 (+ 139us) client_negotiation.cc:240] Negotiation successful
0619 14:46:30.306577 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":107,"threads_started":1}
I20260619 14:46:30.307008 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:30.298250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36257 (local address 127.0.98.62:46827)
0619 14:46:30.298369 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:30.298373 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:30.298604 (+ 231us) server_negotiation.cc:407] Connection header received
0619 14:46:30.298765 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:30.298769 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:30.298817 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:30.298903 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:30.299447 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:30.300314 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:30.301282 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:30.301522 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:30.301641 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:30.302982 (+ 1341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:30.302985 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:30.302987 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:30.303012 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:30.304703 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:30.305201 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:30.305205 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:30.305207 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:30.305303 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:30.305630 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:30.305634 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:30.305635 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:30.305968 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:30.306078 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:30.306467 (+ 389us) server_negotiation.cc:299] Negotiation successful
0619 14:46:30.306610 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880383132139-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880383132139-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 14:46:30.325502 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880383132139-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880383132139-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880383132139-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880383132139-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46827
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880383132139-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:46:30.447880 1859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:46:30.448338 1859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:46:30.448436 1859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:46:30.452893 1859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:46:30.452978 1859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:46:30.453022 1859 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 14:46:30.453055 1859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 14:46:30.458772 1859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880383132139-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880383132139-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880383132139-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880383132139-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:46827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880383132139-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:46:30.460103 1859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:46:30.461208 1859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:46:30.472229 1864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:30.472151 1865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:30.472244 1867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:30.472774 1859 server_base.cc:1034] running on GCE node
I20260619 14:46:30.473280 1859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:46:30.474032 1859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:46:30.475263 1859 hybrid_clock.cc:648] HybridClock initialized: now 1781880390475245 us; error 34 us; skew 500 ppm
Jun 19 14:46:30 dist-test-slave-2sb1 krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880390, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:46:30.478605 1859 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 14:46:30.479650 1859 webserver.cc:460] Webserver started at http://127.0.98.3:34711/ using document root <none> and password file <none>
I20260619 14:46:30.480422 1859 fs_manager.cc:359] Metadata directory not provided
I20260619 14:46:30.480535 1859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:46:30.480835 1859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:46:30.483721 1859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880383132139-392-0/minicluster-data/ts-2/data/instance:
uuid: "3a138c74c83a4052816fc214d30ac87d"
format_stamp: "Formatted at 2026-06-19 14:46:30 on dist-test-slave-2sb1"
server_key: "05d0e850b2d44d3b183aee7b5efbd5ad"
server_key_iv: "a29c84a58aa513e8731aeac8696f494b"
server_key_version: "encryptionkey@0"
I20260619 14:46:30.484426 1859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880383132139-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3a138c74c83a4052816fc214d30ac87d"
format_stamp: "Formatted at 2026-06-19 14:46:30 on dist-test-slave-2sb1"
server_key: "05d0e850b2d44d3b183aee7b5efbd5ad"
server_key_iv: "a29c84a58aa513e8731aeac8696f494b"
server_key_version: "encryptionkey@0"
I20260619 14:46:30.489205 1859 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 14:46:30.492039 1874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:30.493855 1859 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:46:30.494037 1859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880383132139-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880383132139-392-0/minicluster-data/ts-2/wal
uuid: "3a138c74c83a4052816fc214d30ac87d"
format_stamp: "Formatted at 2026-06-19 14:46:30 on dist-test-slave-2sb1"
server_key: "05d0e850b2d44d3b183aee7b5efbd5ad"
server_key_iv: "a29c84a58aa513e8731aeac8696f494b"
server_key_version: "encryptionkey@0"
I20260619 14:46:30.494187 1859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880383132139-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880383132139-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880383132139-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:46:30.505209 1859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:46:30.509044 1859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:46:30.509371 1859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:46:30.511435 1859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:46:30.511540 1859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:30.511626 1859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:46:30.511782 1859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 14:46:30 dist-test-slave-2sb1 krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880390, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:46:30.524725 1859 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41623
I20260619 14:46:30.524744 1989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41623 every 8 connection(s)
I20260619 14:46:30.526219 1859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880383132139-392-0/minicluster-data/ts-2/data/info.pb
I20260619 14:46:30.528558 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:30.512863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44297 (local address 127.0.98.62:46827)
0619 14:46:30.513083 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:30.513087 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:30.514020 (+ 933us) server_negotiation.cc:407] Connection header received
0619 14:46:30.515006 (+ 986us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:30.515010 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:30.515069 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:30.515183 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:30.516911 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:30.517516 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:30.518760 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:30.519223 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:30.519378 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:30.523812 (+ 4434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:30.523816 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:30.523819 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:30.523847 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:30.525910 (+ 2063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:30.526617 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:30.526625 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:30.526630 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:30.526752 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:30.527227 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:30.527230 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:30.527232 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:30.527524 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:30.527621 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:30.528189 (+ 568us) server_negotiation.cc:299] Negotiation successful
0619 14:46:30.528371 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 14:46:30.529577 1883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:30.513348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46827 (local address 127.0.98.3:44297)
0619 14:46:30.513817 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:30.513869 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:30.514782 (+ 913us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:30.515377 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:30.515385 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:30.515990 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:46:30.516731 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:30.516748 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:30.517653 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:30.517657 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:30.518625 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:30.518638 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:30.519552 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:30.519559 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:30.519836 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:30.520588 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:46:30.520608 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:46:30.523500 (+ 2892us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:46:30.526067 (+ 2567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:30.526073 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:30.526085 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:30.526466 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:30.526882 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:30.526886 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:30.526888 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:30.527090 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:30.527634 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:46:30.527641 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:46:30.527918 (+ 277us) client_negotiation.cc:769] Sending connection context
0619 14:46:30.528097 (+ 179us) client_negotiation.cc:240] Negotiation successful
0619 14:46:30.528564 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":138,"threads_started":1}
I20260619 14:46:30.531365 1990 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46827
I20260619 14:46:30.531544 1990 heartbeater.cc:464] Registering TS with master...
I20260619 14:46:30.532501 1990 heartbeater.cc:511] Master 127.0.98.62:46827 requested a full tablet report, sending...
I20260619 14:46:30.533749 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 1859
I20260619 14:46:30.533838 1507 ts_manager.cc:194] Registered new tserver with Master: 3a138c74c83a4052816fc214d30ac87d (127.0.98.3:41623)
I20260619 14:46:30.533892 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880383132139-392-0/minicluster-data/ts-2/wal/instance
I20260619 14:46:30.534332 392 external_mini_cluster.cc:1428] Setting key 2ffac27a98fe67113210c45174d1ff87
I20260619 14:46:30.534529 1507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44297
I20260619 14:46:30.537055 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:46:30.544255 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:30.535056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55989 (local address 127.0.98.62:46827)
0619 14:46:30.535200 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:30.535204 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:30.535476 (+ 272us) server_negotiation.cc:407] Connection header received
0619 14:46:30.535552 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:30.535559 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:30.535611 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:30.535793 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:30.536526 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:30.537302 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:30.538596 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:30.538867 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:30.538966 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:30.540574 (+ 1608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:30.540581 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:30.540584 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:30.540616 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:30.542501 (+ 1885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:30.542981 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:30.542986 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:30.542988 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:30.543079 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:30.543372 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:30.543377 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:30.543378 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:30.543573 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:30.543698 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:30.543960 (+ 262us) server_negotiation.cc:299] Negotiation successful
0619 14:46:30.544072 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 14:46:30.544283 1996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:30.534960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46827 (local address 127.0.98.3:55989)
0619 14:46:30.535322 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:30.535338 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:30.535452 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:30.535945 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:30.535947 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:30.536238 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:46:30.536378 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:30.536387 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:30.537450 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:30.537454 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:30.538432 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:30.538448 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:30.539027 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:30.539034 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:30.539169 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:30.539828 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:46:30.539850 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:46:30.540266 (+ 416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:46:30.542637 (+ 2371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:30.542640 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:30.542643 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:30.542871 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:30.543182 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:30.543185 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:30.543188 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:30.543280 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:30.543671 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:46:30.543674 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:46:30.543789 (+ 115us) client_negotiation.cc:769] Sending connection context
0619 14:46:30.543905 (+ 116us) client_negotiation.cc:240] Negotiation successful
0619 14:46:30.544077 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":168,"threads_started":1}
Jun 19 14:46:30 dist-test-slave-2sb1 krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880390, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 14:46:30 dist-test-slave-2sb1 krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880390, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:46:30.568794 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:30.557335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58246 (local address 127.0.98.62:46827)
0619 14:46:30.557508 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:30.557516 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:30.557537 (+ 21us) server_negotiation.cc:407] Connection header received
0619 14:46:30.557665 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:30.557668 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:30.557713 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:30.557824 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:30.558641 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:30.559376 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:30.560450 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:30.560819 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:30.560957 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:30.564224 (+ 3267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:30.564228 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:30.564231 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:30.564284 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:30.566434 (+ 2150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:30.567135 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:30.567140 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:30.567141 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:30.567251 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:30.567605 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:30.567608 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:30.567610 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:30.567843 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:30.567961 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:30.568393 (+ 432us) server_negotiation.cc:299] Negotiation successful
0619 14:46:30.568511 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 14:46:30.572845 1507 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58246:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:46:30.577176 1507 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:46:30.596468 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:30.588781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46190 (local address 127.0.98.2:38555)
0619 14:46:30.589652 (+ 871us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:30.589666 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:30.589695 (+ 29us) server_negotiation.cc:407] Connection header received
0619 14:46:30.589782 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:30.589797 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:30.590052 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:30.590241 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:46:30.590902 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:30.592234 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:30.594251 (+ 2017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:30.595252 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:30.595377 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:30.595564 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:30.596180 (+ 616us) server_negotiation.cc:299] Negotiation successful
0619 14:46:30.596292 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":758,"thread_start_us":139,"threads_started":1}
I20260619 14:46:30.596457 2007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:30.588510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38555 (local address 127.0.0.1:46190)
0619 14:46:30.589124 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:30.589184 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:30.589335 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:30.590474 (+ 1139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:30.590481 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:30.590505 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:46:30.590753 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:30.590761 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:30.592382 (+ 1621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:30.592387 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:30.594111 (+ 1724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:30.594122 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:30.595660 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:30.595667 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:30.595869 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:30.595920 (+ 51us) client_negotiation.cc:769] Sending connection context
0619 14:46:30.596127 (+ 207us) client_negotiation.cc:240] Negotiation successful
0619 14:46:30.596289 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"mutex_wait_us":195,"thread_start_us":79,"threads_started":1}
I20260619 14:46:30.597126 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:30.588847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41623 (local address 127.0.0.1:40842)
0619 14:46:30.589274 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:30.589287 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:30.589358 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:30.590599 (+ 1241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:30.590602 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:30.590615 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:46:30.590759 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:30.590765 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:30.592628 (+ 1863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:30.592635 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:30.594451 (+ 1816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:30.594463 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:30.595737 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:30.595741 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:30.596008 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:30.596048 (+ 40us) client_negotiation.cc:769] Sending connection context
0619 14:46:30.596169 (+ 121us) client_negotiation.cc:240] Negotiation successful
0619 14:46:30.596616 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"mutex_wait_us":624,"thread_start_us":95,"threads_started":1}
I20260619 14:46:30.597306 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:30.589613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33672 (local address 127.0.98.1:46285)
0619 14:46:30.590620 (+ 1007us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:30.590627 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:30.590651 (+ 24us) server_negotiation.cc:407] Connection header received
0619 14:46:30.590733 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:30.590739 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:30.590973 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:30.591153 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:46:30.591407 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:30.592287 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:30.594700 (+ 2413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:30.595595 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:30.595743 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:30.595899 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:30.596999 (+ 1100us) server_negotiation.cc:299] Negotiation successful
0619 14:46:30.597154 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":885,"thread_start_us":73,"threads_started":1}
I20260619 14:46:30.597713 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:30.589617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40842 (local address 127.0.98.3:41623)
0619 14:46:30.590109 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:30.590126 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:30.590153 (+ 27us) server_negotiation.cc:407] Connection header received
0619 14:46:30.590234 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:30.590242 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:30.590445 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:30.590636 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:46:30.591322 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:30.592531 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:30.594644 (+ 2113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:30.595603 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:30.595735 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:30.596031 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:30.597472 (+ 1441us) server_negotiation.cc:299] Negotiation successful
0619 14:46:30.597585 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":128,"threads_started":1}
I20260619 14:46:30.598601 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:30.589308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46285 (local address 127.0.0.1:33672)
0619 14:46:30.590073 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:30.590090 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:30.590200 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:30.591137 (+ 937us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:30.591140 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:30.591161 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:46:30.591271 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:30.591276 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:30.592425 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:30.592429 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:30.594451 (+ 2022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:30.594463 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:30.595847 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:30.595853 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:30.596307 (+ 454us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:30.596329 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 14:46:30.596403 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 14:46:30.596482 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"mutex_wait_us":489,"thread_start_us":96,"threads_started":1}
I20260619 14:46:30.600816 1924 tablet_service.cc:1467] Processing CreateTablet for tablet a40a76e4292f439d98589fce62ce5a33 (DEFAULT_TABLE table=test-table [id=d3603b8ea7ed49c49ac06d97802455c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:46:30.600819 1784 tablet_service.cc:1467] Processing CreateTablet for tablet a40a76e4292f439d98589fce62ce5a33 (DEFAULT_TABLE table=test-table [id=d3603b8ea7ed49c49ac06d97802455c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:46:30.601953 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a40a76e4292f439d98589fce62ce5a33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:46:30.601971 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a40a76e4292f439d98589fce62ce5a33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:46:30.601632 1644 tablet_service.cc:1467] Processing CreateTablet for tablet a40a76e4292f439d98589fce62ce5a33 (DEFAULT_TABLE table=test-table [id=d3603b8ea7ed49c49ac06d97802455c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:46:30.602895 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a40a76e4292f439d98589fce62ce5a33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:46:30.610252 2013 tablet_bootstrap.cc:492] T a40a76e4292f439d98589fce62ce5a33 P 3a138c74c83a4052816fc214d30ac87d: Bootstrap starting.
I20260619 14:46:30.612107 2014 tablet_bootstrap.cc:492] T a40a76e4292f439d98589fce62ce5a33 P 5e5d18da021245748c1655fe7fed35b1: Bootstrap starting.
I20260619 14:46:30.613081 2015 tablet_bootstrap.cc:492] T a40a76e4292f439d98589fce62ce5a33 P e4742b727bc24e89ad40347bfc4a8942: Bootstrap starting.
I20260619 14:46:30.613656 2013 tablet_bootstrap.cc:654] T a40a76e4292f439d98589fce62ce5a33 P 3a138c74c83a4052816fc214d30ac87d: Neither blocks nor log segments found. Creating new log.
I20260619 14:46:30.614661 2013 log.cc:824] T a40a76e4292f439d98589fce62ce5a33 P 3a138c74c83a4052816fc214d30ac87d: Log is configured to *not* fsync() on all Append() calls
I20260619 14:46:30.614833 2014 tablet_bootstrap.cc:654] T a40a76e4292f439d98589fce62ce5a33 P 5e5d18da021245748c1655fe7fed35b1: Neither blocks nor log segments found. Creating new log.
I20260619 14:46:30.615372 2015 tablet_bootstrap.cc:654] T a40a76e4292f439d98589fce62ce5a33 P e4742b727bc24e89ad40347bfc4a8942: Neither blocks nor log segments found. Creating new log.
I20260619 14:46:30.615674 2014 log.cc:824] T a40a76e4292f439d98589fce62ce5a33 P 5e5d18da021245748c1655fe7fed35b1: Log is configured to *not* fsync() on all Append() calls
I20260619 14:46:30.616317 2015 log.cc:824] T a40a76e4292f439d98589fce62ce5a33 P e4742b727bc24e89ad40347bfc4a8942: Log is configured to *not* fsync() on all Append() calls
I20260619 14:46:30.617370 2013 tablet_bootstrap.cc:492] T a40a76e4292f439d98589fce62ce5a33 P 3a138c74c83a4052816fc214d30ac87d: No bootstrap required, opened a new log
I20260619 14:46:30.617650 2013 ts_tablet_manager.cc:1397] T a40a76e4292f439d98589fce62ce5a33 P 3a138c74c83a4052816fc214d30ac87d: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260619 14:46:30.618058 2014 tablet_bootstrap.cc:492] T a40a76e4292f439d98589fce62ce5a33 P 5e5d18da021245748c1655fe7fed35b1: No bootstrap required, opened a new log
I20260619 14:46:30.618253 2014 ts_tablet_manager.cc:1397] T a40a76e4292f439d98589fce62ce5a33 P 5e5d18da021245748c1655fe7fed35b1: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 14:46:30.618793 2015 tablet_bootstrap.cc:492] T a40a76e4292f439d98589fce62ce5a33 P e4742b727bc24e89ad40347bfc4a8942: No bootstrap required, opened a new log
I20260619 14:46:30.619027 2015 ts_tablet_manager.cc:1397] T a40a76e4292f439d98589fce62ce5a33 P e4742b727bc24e89ad40347bfc4a8942: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 14:46:30.622524 2013 raft_consensus.cc:348] T a40a76e4292f439d98589fce62ce5a33 P 3a138c74c83a4052816fc214d30ac87d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4742b727bc24e89ad40347bfc4a8942" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38555 } } peers { permanent_uuid: "3a138c74c83a4052816fc214d30ac87d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41623 } } peers { permanent_uuid: "5e5d18da021245748c1655fe7fed35b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46285 } }
I20260619 14:46:30.622650 2015 raft_consensus.cc:348] T a40a76e4292f439d98589fce62ce5a33 P e4742b727bc24e89ad40347bfc4a8942 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4742b727bc24e89ad40347bfc4a8942" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38555 } } peers { permanent_uuid: "3a138c74c83a4052816fc214d30ac87d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41623 } } peers { permanent_uuid: "5e5d18da021245748c1655fe7fed35b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46285 } }
I20260619 14:46:30.622871 2013 raft_consensus.cc:374] T a40a76e4292f439d98589fce62ce5a33 P 3a138c74c83a4052816fc214d30ac87d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:46:30.622900 2015 raft_consensus.cc:374] T a40a76e4292f439d98589fce62ce5a33 P e4742b727bc24e89ad40347bfc4a8942 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:46:30.622943 2013 raft_consensus.cc:729] T a40a76e4292f439d98589fce62ce5a33 P 3a138c74c83a4052816fc214d30ac87d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a138c74c83a4052816fc214d30ac87d, State: Initialized, Role: FOLLOWER
I20260619 14:46:30.622954 2015 raft_consensus.cc:729] T a40a76e4292f439d98589fce62ce5a33 P e4742b727bc24e89ad40347bfc4a8942 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4742b727bc24e89ad40347bfc4a8942, State: Initialized, Role: FOLLOWER
I20260619 14:46:30.623229 2014 raft_consensus.cc:348] T a40a76e4292f439d98589fce62ce5a33 P 5e5d18da021245748c1655fe7fed35b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4742b727bc24e89ad40347bfc4a8942" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38555 } } peers { permanent_uuid: "3a138c74c83a4052816fc214d30ac87d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41623 } } peers { permanent_uuid: "5e5d18da021245748c1655fe7fed35b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46285 } }
I20260619 14:46:30.623493 2014 raft_consensus.cc:374] T a40a76e4292f439d98589fce62ce5a33 P 5e5d18da021245748c1655fe7fed35b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:46:30.623432 2015 consensus_queue.cc:260] T a40a76e4292f439d98589fce62ce5a33 P e4742b727bc24e89ad40347bfc4a8942 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4742b727bc24e89ad40347bfc4a8942" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38555 } } peers { permanent_uuid: "3a138c74c83a4052816fc214d30ac87d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41623 } } peers { permanent_uuid: "5e5d18da021245748c1655fe7fed35b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46285 } }
I20260619 14:46:30.623442 2013 consensus_queue.cc:260] T a40a76e4292f439d98589fce62ce5a33 P 3a138c74c83a4052816fc214d30ac87d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4742b727bc24e89ad40347bfc4a8942" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38555 } } peers { permanent_uuid: "3a138c74c83a4052816fc214d30ac87d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41623 } } peers { permanent_uuid: "5e5d18da021245748c1655fe7fed35b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46285 } }
I20260619 14:46:30.623561 2014 raft_consensus.cc:729] T a40a76e4292f439d98589fce62ce5a33 P 5e5d18da021245748c1655fe7fed35b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e5d18da021245748c1655fe7fed35b1, State: Initialized, Role: FOLLOWER
I20260619 14:46:30.623992 2014 consensus_queue.cc:260] T a40a76e4292f439d98589fce62ce5a33 P 5e5d18da021245748c1655fe7fed35b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4742b727bc24e89ad40347bfc4a8942" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38555 } } peers { permanent_uuid: "3a138c74c83a4052816fc214d30ac87d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41623 } } peers { permanent_uuid: "5e5d18da021245748c1655fe7fed35b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46285 } }
I20260619 14:46:30.624608 2015 ts_tablet_manager.cc:1428] T a40a76e4292f439d98589fce62ce5a33 P e4742b727bc24e89ad40347bfc4a8942: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260619 14:46:30.624758 1710 heartbeater.cc:503] Master 127.0.98.62:46827 was elected leader, sending a full tablet report...
I20260619 14:46:30.624930 2014 ts_tablet_manager.cc:1428] T a40a76e4292f439d98589fce62ce5a33 P 5e5d18da021245748c1655fe7fed35b1: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260619 14:46:30.626087 1850 heartbeater.cc:503] Master 127.0.98.62:46827 was elected leader, sending a full tablet report...
I20260619 14:46:30.627345 2013 ts_tablet_manager.cc:1428] T a40a76e4292f439d98589fce62ce5a33 P 3a138c74c83a4052816fc214d30ac87d: Time spent starting tablet: real 0.010s user 0.008s sys 0.000s
I20260619 14:46:30.627851 1990 heartbeater.cc:503] Master 127.0.98.62:46827 was elected leader, sending a full tablet report...
I20260619 14:46:30.627832 2021 raft_consensus.cc:482] T a40a76e4292f439d98589fce62ce5a33 P 5e5d18da021245748c1655fe7fed35b1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:46:30.628459 2021 raft_consensus.cc:504] T a40a76e4292f439d98589fce62ce5a33 P 5e5d18da021245748c1655fe7fed35b1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4742b727bc24e89ad40347bfc4a8942" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38555 } } peers { permanent_uuid: "3a138c74c83a4052816fc214d30ac87d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41623 } } peers { permanent_uuid: "5e5d18da021245748c1655fe7fed35b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46285 } }
I20260619 14:46:30.631084 2021 leader_election.cc:288] T a40a76e4292f439d98589fce62ce5a33 P 5e5d18da021245748c1655fe7fed35b1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4742b727bc24e89ad40347bfc4a8942 (127.0.98.2:38555), 3a138c74c83a4052816fc214d30ac87d (127.0.98.3:41623)
I20260619 14:46:30.635623 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:30.631082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57059 (local address 127.0.98.2:38555)
0619 14:46:30.631455 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:30.631460 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:30.631481 (+ 21us) server_negotiation.cc:407] Connection header received
0619 14:46:30.631661 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:30.631667 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:30.631724 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:30.631798 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:46:30.632246 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:30.633069 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:30.634563 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:30.635041 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:30.635141 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:30.635206 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:30.635424 (+ 218us) server_negotiation.cc:299] Negotiation successful
0619 14:46:30.635500 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279}
I20260619 14:46:30.635650 2023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:30.630727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38555 (local address 127.0.98.1:57059)
0619 14:46:30.631409 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:30.631425 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:30.631529 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:30.631924 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:30.631927 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:30.631941 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:46:30.632103 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:30.632114 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:30.633174 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:30.633178 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:30.634419 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:30.634431 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:30.635159 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:30.635163 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:30.635278 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:30.635297 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 14:46:30.635394 (+ 97us) client_negotiation.cc:240] Negotiation successful
0619 14:46:30.635474 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"mutex_wait_us":24,"thread_start_us":129,"threads_started":1}
I20260619 14:46:30.635807 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:30.631136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35383 (local address 127.0.98.3:41623)
0619 14:46:30.631325 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:30.631331 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:30.631861 (+ 530us) server_negotiation.cc:407] Connection header received
0619 14:46:30.632153 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:30.632158 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:30.632212 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:30.632294 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:46:30.632583 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:30.633247 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:30.634590 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:30.635172 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:30.635296 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:30.635360 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:30.635595 (+ 235us) server_negotiation.cc:299] Negotiation successful
0619 14:46:30.635673 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260619 14:46:30.635830 2022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:30.630727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41623 (local address 127.0.98.1:35383)
0619 14:46:30.631783 (+ 1056us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:30.631798 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:30.631922 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:30.632318 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:30.632321 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:30.632344 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:46:30.632452 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:30.632458 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:30.633379 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:30.633384 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:30.634450 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:30.634462 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:30.635313 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:30.635317 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:30.635449 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:30.635475 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 14:46:30.635564 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 14:46:30.635653 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":974,"mutex_wait_us":134,"thread_start_us":86,"threads_started":1}
I20260619 14:46:30.636540 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a40a76e4292f439d98589fce62ce5a33" candidate_uuid: "5e5d18da021245748c1655fe7fed35b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a138c74c83a4052816fc214d30ac87d" is_pre_election: true
I20260619 14:46:30.636714 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a40a76e4292f439d98589fce62ce5a33" candidate_uuid: "5e5d18da021245748c1655fe7fed35b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4742b727bc24e89ad40347bfc4a8942" is_pre_election: true
I20260619 14:46:30.636968 1944 raft_consensus.cc:2454] T a40a76e4292f439d98589fce62ce5a33 P 3a138c74c83a4052816fc214d30ac87d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e5d18da021245748c1655fe7fed35b1 in term 0.
I20260619 14:46:30.637012 1804 raft_consensus.cc:2454] T a40a76e4292f439d98589fce62ce5a33 P e4742b727bc24e89ad40347bfc4a8942 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e5d18da021245748c1655fe7fed35b1 in term 0.
I20260619 14:46:30.637490 1596 leader_election.cc:302] T a40a76e4292f439d98589fce62ce5a33 P 5e5d18da021245748c1655fe7fed35b1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e5d18da021245748c1655fe7fed35b1, e4742b727bc24e89ad40347bfc4a8942; no voters:
I20260619 14:46:30.637785 2021 raft_consensus.cc:2781] T a40a76e4292f439d98589fce62ce5a33 P 5e5d18da021245748c1655fe7fed35b1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:46:30.637877 2021 raft_consensus.cc:482] T a40a76e4292f439d98589fce62ce5a33 P 5e5d18da021245748c1655fe7fed35b1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:46:30.637936 2021 raft_consensus.cc:3037] T a40a76e4292f439d98589fce62ce5a33 P 5e5d18da021245748c1655fe7fed35b1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:46:30.639042 2021 raft_consensus.cc:504] T a40a76e4292f439d98589fce62ce5a33 P 5e5d18da021245748c1655fe7fed35b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4742b727bc24e89ad40347bfc4a8942" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38555 } } peers { permanent_uuid: "3a138c74c83a4052816fc214d30ac87d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41623 } } peers { permanent_uuid: "5e5d18da021245748c1655fe7fed35b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46285 } }
I20260619 14:46:30.639470 2021 leader_election.cc:288] T a40a76e4292f439d98589fce62ce5a33 P 5e5d18da021245748c1655fe7fed35b1 [CANDIDATE]: Term 1 election: Requested vote from peers e4742b727bc24e89ad40347bfc4a8942 (127.0.98.2:38555), 3a138c74c83a4052816fc214d30ac87d (127.0.98.3:41623)
I20260619 14:46:30.639904 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a40a76e4292f439d98589fce62ce5a33" candidate_uuid: "5e5d18da021245748c1655fe7fed35b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4742b727bc24e89ad40347bfc4a8942"
I20260619 14:46:30.640086 1804 raft_consensus.cc:3037] T a40a76e4292f439d98589fce62ce5a33 P e4742b727bc24e89ad40347bfc4a8942 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:46:30.640870 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a40a76e4292f439d98589fce62ce5a33" candidate_uuid: "5e5d18da021245748c1655fe7fed35b1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a138c74c83a4052816fc214d30ac87d"
I20260619 14:46:30.640997 1944 raft_consensus.cc:3037] T a40a76e4292f439d98589fce62ce5a33 P 3a138c74c83a4052816fc214d30ac87d [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:46:30.641413 1804 raft_consensus.cc:2454] T a40a76e4292f439d98589fce62ce5a33 P e4742b727bc24e89ad40347bfc4a8942 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e5d18da021245748c1655fe7fed35b1 in term 1.
I20260619 14:46:30.641805 1596 leader_election.cc:302] T a40a76e4292f439d98589fce62ce5a33 P 5e5d18da021245748c1655fe7fed35b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e5d18da021245748c1655fe7fed35b1, e4742b727bc24e89ad40347bfc4a8942; no voters:
I20260619 14:46:30.642000 2021 raft_consensus.cc:2781] T a40a76e4292f439d98589fce62ce5a33 P 5e5d18da021245748c1655fe7fed35b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:46:30.642266 2021 raft_consensus.cc:686] T a40a76e4292f439d98589fce62ce5a33 P 5e5d18da021245748c1655fe7fed35b1 [term 1 LEADER]: Becoming Leader. State: Replica: 5e5d18da021245748c1655fe7fed35b1, State: Running, Role: LEADER
I20260619 14:46:30.642516 1944 raft_consensus.cc:2454] T a40a76e4292f439d98589fce62ce5a33 P 3a138c74c83a4052816fc214d30ac87d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e5d18da021245748c1655fe7fed35b1 in term 1.
I20260619 14:46:30.642705 2021 consensus_queue.cc:237] T a40a76e4292f439d98589fce62ce5a33 P 5e5d18da021245748c1655fe7fed35b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4742b727bc24e89ad40347bfc4a8942" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38555 } } peers { permanent_uuid: "3a138c74c83a4052816fc214d30ac87d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41623 } } peers { permanent_uuid: "5e5d18da021245748c1655fe7fed35b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46285 } }
I20260619 14:46:30.646304 1506 catalog_manager.cc:5495] T a40a76e4292f439d98589fce62ce5a33 P 5e5d18da021245748c1655fe7fed35b1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e5d18da021245748c1655fe7fed35b1 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "5e5d18da021245748c1655fe7fed35b1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4742b727bc24e89ad40347bfc4a8942" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38555 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a138c74c83a4052816fc214d30ac87d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e5d18da021245748c1655fe7fed35b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46285 } health_report { overall_health: HEALTHY } } }
I20260619 14:46:30.677999 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:30.673898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33684 (local address 127.0.98.1:46285)
0619 14:46:30.674045 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:30.674049 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:30.674080 (+ 31us) server_negotiation.cc:407] Connection header received
0619 14:46:30.674378 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:30.674381 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:30.674441 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:30.674548 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:46:30.674876 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:30.675404 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:30.676333 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:30.676565 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:30.676729 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:30.677099 (+ 370us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:46:30.677543 (+ 444us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:46:30.677671 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:30.677767 (+ 96us) server_negotiation.cc:299] Negotiation successful
0619 14:46:30.677830 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 14:46:30.686029 1944 raft_consensus.cc:1261] T a40a76e4292f439d98589fce62ce5a33 P 3a138c74c83a4052816fc214d30ac87d [term 1 FOLLOWER]: Refusing update from remote peer 5e5d18da021245748c1655fe7fed35b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:46:30.686029 1804 raft_consensus.cc:1261] T a40a76e4292f439d98589fce62ce5a33 P e4742b727bc24e89ad40347bfc4a8942 [term 1 FOLLOWER]: Refusing update from remote peer 5e5d18da021245748c1655fe7fed35b1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:46:30.687320 2024 consensus_queue.cc:1035] T a40a76e4292f439d98589fce62ce5a33 P 5e5d18da021245748c1655fe7fed35b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4742b727bc24e89ad40347bfc4a8942" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:46:30.687974 2021 consensus_queue.cc:1035] T a40a76e4292f439d98589fce62ce5a33 P 5e5d18da021245748c1655fe7fed35b1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a138c74c83a4052816fc214d30ac87d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:46:30.698019 2029 mvcc.cc:205] Tried to move back new op lower bound from 7298582080238477312 to 7298582080075464704. Current Snapshot: MvccSnapshot[applied={T|T < 7298582080238477312}]
I20260619 14:46:30.698354 2028 mvcc.cc:205] Tried to move back new op lower bound from 7298582080238477312 to 7298582080075464704. Current Snapshot: MvccSnapshot[applied={T|T < 7298582080238477312}]
I20260619 14:46:30.711644 1506 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58246:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:46:30.711838 1506 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58246:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:46:30.715044 1506 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 11ee01cd7a424422a7e03f596fd13d49: Sending DeleteTablet for 3 replicas of tablet a40a76e4292f439d98589fce62ce5a33
I20260619 14:46:30.716009 1784 tablet_service.cc:1514] Processing DeleteTablet for tablet a40a76e4292f439d98589fce62ce5a33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:46:30 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46190
I20260619 14:46:30.716009 1924 tablet_service.cc:1514] Processing DeleteTablet for tablet a40a76e4292f439d98589fce62ce5a33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:46:30 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40842
I20260619 14:46:30.716171 1644 tablet_service.cc:1514] Processing DeleteTablet for tablet a40a76e4292f439d98589fce62ce5a33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:46:30 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33672
I20260619 14:46:30.716630 2038 tablet_replica.cc:330] T a40a76e4292f439d98589fce62ce5a33 P 3a138c74c83a4052816fc214d30ac87d: stopping tablet replica
I20260619 14:46:30.716638 2037 tablet_replica.cc:330] T a40a76e4292f439d98589fce62ce5a33 P e4742b727bc24e89ad40347bfc4a8942: stopping tablet replica
I20260619 14:46:30.717025 2038 raft_consensus.cc:2229] T a40a76e4292f439d98589fce62ce5a33 P 3a138c74c83a4052816fc214d30ac87d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:46:30.717386 2038 raft_consensus.cc:2258] T a40a76e4292f439d98589fce62ce5a33 P 3a138c74c83a4052816fc214d30ac87d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:46:30.717303 2039 tablet_replica.cc:330] T a40a76e4292f439d98589fce62ce5a33 P 5e5d18da021245748c1655fe7fed35b1: stopping tablet replica
I20260619 14:46:30.717720 2037 raft_consensus.cc:2229] T a40a76e4292f439d98589fce62ce5a33 P e4742b727bc24e89ad40347bfc4a8942 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:46:30.717978 2039 raft_consensus.cc:2229] T a40a76e4292f439d98589fce62ce5a33 P 5e5d18da021245748c1655fe7fed35b1 [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:46:30.718030 2037 raft_consensus.cc:2258] T a40a76e4292f439d98589fce62ce5a33 P e4742b727bc24e89ad40347bfc4a8942 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:46:30.717940 1506 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58246:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:46:30.718400 1506 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:46:30.719614 2037 ts_tablet_manager.cc:1905] T a40a76e4292f439d98589fce62ce5a33 P e4742b727bc24e89ad40347bfc4a8942: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:46:30.720414 2038 ts_tablet_manager.cc:1905] T a40a76e4292f439d98589fce62ce5a33 P 3a138c74c83a4052816fc214d30ac87d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:46:30.721241 2039 raft_consensus.cc:2258] T a40a76e4292f439d98589fce62ce5a33 P 5e5d18da021245748c1655fe7fed35b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:46:30.722613 2037 ts_tablet_manager.cc:1918] T a40a76e4292f439d98589fce62ce5a33 P e4742b727bc24e89ad40347bfc4a8942: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:46:30.722730 2037 log.cc:1196] T a40a76e4292f439d98589fce62ce5a33 P e4742b727bc24e89ad40347bfc4a8942: Deleting WAL directory at /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880383132139-392-0/minicluster-data/ts-1/wal/wals/a40a76e4292f439d98589fce62ce5a33
I20260619 14:46:30.723069 2037 ts_tablet_manager.cc:1939] T a40a76e4292f439d98589fce62ce5a33 P e4742b727bc24e89ad40347bfc4a8942: Deleting consensus metadata
I20260619 14:46:30.724377 1492 catalog_manager.cc:4841] TS e4742b727bc24e89ad40347bfc4a8942 (127.0.98.2:38555): tablet a40a76e4292f439d98589fce62ce5a33 (table test-table [id=d3603b8ea7ed49c49ac06d97802455c6]) successfully deleted
I20260619 14:46:30.724433 1784 tablet_service.cc:1467] Processing CreateTablet for tablet a454436a865e42a8b2fdeaccdb2b28f9 (DEFAULT_TABLE table=test-table [id=d01f16fa84aa4068a362f45526cd0e19]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:46:30.724740 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a454436a865e42a8b2fdeaccdb2b28f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:46:30.725394 1644 tablet_service.cc:1467] Processing CreateTablet for tablet a454436a865e42a8b2fdeaccdb2b28f9 (DEFAULT_TABLE table=test-table [id=d01f16fa84aa4068a362f45526cd0e19]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:46:30.725654 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a454436a865e42a8b2fdeaccdb2b28f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:46:30.725863 2039 ts_tablet_manager.cc:1905] T a40a76e4292f439d98589fce62ce5a33 P 5e5d18da021245748c1655fe7fed35b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:46:30.727413 2015 tablet_bootstrap.cc:492] T a454436a865e42a8b2fdeaccdb2b28f9 P e4742b727bc24e89ad40347bfc4a8942: Bootstrap starting.
I20260619 14:46:30.727751 2038 ts_tablet_manager.cc:1918] T a40a76e4292f439d98589fce62ce5a33 P 3a138c74c83a4052816fc214d30ac87d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:46:30.727981 2038 log.cc:1196] T a40a76e4292f439d98589fce62ce5a33 P 3a138c74c83a4052816fc214d30ac87d: Deleting WAL directory at /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880383132139-392-0/minicluster-data/ts-2/wal/wals/a40a76e4292f439d98589fce62ce5a33
I20260619 14:46:30.728386 2038 ts_tablet_manager.cc:1939] T a40a76e4292f439d98589fce62ce5a33 P 3a138c74c83a4052816fc214d30ac87d: Deleting consensus metadata
I20260619 14:46:30.728452 2015 tablet_bootstrap.cc:654] T a454436a865e42a8b2fdeaccdb2b28f9 P e4742b727bc24e89ad40347bfc4a8942: Neither blocks nor log segments found. Creating new log.
I20260619 14:46:30.728758 2014 tablet_bootstrap.cc:492] T a454436a865e42a8b2fdeaccdb2b28f9 P 5e5d18da021245748c1655fe7fed35b1: Bootstrap starting.
I20260619 14:46:30.728531 1924 tablet_service.cc:1467] Processing CreateTablet for tablet a454436a865e42a8b2fdeaccdb2b28f9 (DEFAULT_TABLE table=test-table [id=d01f16fa84aa4068a362f45526cd0e19]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:46:30.729228 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a454436a865e42a8b2fdeaccdb2b28f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:46:30.729790 2014 tablet_bootstrap.cc:654] T a454436a865e42a8b2fdeaccdb2b28f9 P 5e5d18da021245748c1655fe7fed35b1: Neither blocks nor log segments found. Creating new log.
I20260619 14:46:30.730293 2015 tablet_bootstrap.cc:492] T a454436a865e42a8b2fdeaccdb2b28f9 P e4742b727bc24e89ad40347bfc4a8942: No bootstrap required, opened a new log
I20260619 14:46:30.730373 2015 ts_tablet_manager.cc:1397] T a454436a865e42a8b2fdeaccdb2b28f9 P e4742b727bc24e89ad40347bfc4a8942: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260619 14:46:30.730839 2015 raft_consensus.cc:348] T a454436a865e42a8b2fdeaccdb2b28f9 P e4742b727bc24e89ad40347bfc4a8942 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5d18da021245748c1655fe7fed35b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46285 } } peers { permanent_uuid: "e4742b727bc24e89ad40347bfc4a8942" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38555 } } peers { permanent_uuid: "3a138c74c83a4052816fc214d30ac87d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41623 } }
I20260619 14:46:30.730957 2015 raft_consensus.cc:374] T a454436a865e42a8b2fdeaccdb2b28f9 P e4742b727bc24e89ad40347bfc4a8942 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:46:30.730999 2015 raft_consensus.cc:729] T a454436a865e42a8b2fdeaccdb2b28f9 P e4742b727bc24e89ad40347bfc4a8942 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4742b727bc24e89ad40347bfc4a8942, State: Initialized, Role: FOLLOWER
I20260619 14:46:30.731125 2015 consensus_queue.cc:260] T a454436a865e42a8b2fdeaccdb2b28f9 P e4742b727bc24e89ad40347bfc4a8942 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5d18da021245748c1655fe7fed35b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46285 } } peers { permanent_uuid: "e4742b727bc24e89ad40347bfc4a8942" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38555 } } peers { permanent_uuid: "3a138c74c83a4052816fc214d30ac87d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41623 } }
I20260619 14:46:30.731431 2015 ts_tablet_manager.cc:1428] T a454436a865e42a8b2fdeaccdb2b28f9 P e4742b727bc24e89ad40347bfc4a8942: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260619 14:46:30.732029 2013 tablet_bootstrap.cc:492] T a454436a865e42a8b2fdeaccdb2b28f9 P 3a138c74c83a4052816fc214d30ac87d: Bootstrap starting.
I20260619 14:46:30.732066 2014 tablet_bootstrap.cc:492] T a454436a865e42a8b2fdeaccdb2b28f9 P 5e5d18da021245748c1655fe7fed35b1: No bootstrap required, opened a new log
I20260619 14:46:30.732151 2014 ts_tablet_manager.cc:1397] T a454436a865e42a8b2fdeaccdb2b28f9 P 5e5d18da021245748c1655fe7fed35b1: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 14:46:30.732589 2014 raft_consensus.cc:348] T a454436a865e42a8b2fdeaccdb2b28f9 P 5e5d18da021245748c1655fe7fed35b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5d18da021245748c1655fe7fed35b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46285 } } peers { permanent_uuid: "e4742b727bc24e89ad40347bfc4a8942" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38555 } } peers { permanent_uuid: "3a138c74c83a4052816fc214d30ac87d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41623 } }
I20260619 14:46:30.732760 2014 raft_consensus.cc:374] T a454436a865e42a8b2fdeaccdb2b28f9 P 5e5d18da021245748c1655fe7fed35b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:46:30.732803 2014 raft_consensus.cc:729] T a454436a865e42a8b2fdeaccdb2b28f9 P 5e5d18da021245748c1655fe7fed35b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e5d18da021245748c1655fe7fed35b1, State: Initialized, Role: FOLLOWER
I20260619 14:46:30.732932 2014 consensus_queue.cc:260] T a454436a865e42a8b2fdeaccdb2b28f9 P 5e5d18da021245748c1655fe7fed35b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5d18da021245748c1655fe7fed35b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46285 } } peers { permanent_uuid: "e4742b727bc24e89ad40347bfc4a8942" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38555 } } peers { permanent_uuid: "3a138c74c83a4052816fc214d30ac87d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41623 } }
I20260619 14:46:30.733121 2013 tablet_bootstrap.cc:654] T a454436a865e42a8b2fdeaccdb2b28f9 P 3a138c74c83a4052816fc214d30ac87d: Neither blocks nor log segments found. Creating new log.
I20260619 14:46:30.733240 2014 ts_tablet_manager.cc:1428] T a454436a865e42a8b2fdeaccdb2b28f9 P 5e5d18da021245748c1655fe7fed35b1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:46:30.734457 2013 tablet_bootstrap.cc:492] T a454436a865e42a8b2fdeaccdb2b28f9 P 3a138c74c83a4052816fc214d30ac87d: No bootstrap required, opened a new log
I20260619 14:46:30.734565 2013 ts_tablet_manager.cc:1397] T a454436a865e42a8b2fdeaccdb2b28f9 P 3a138c74c83a4052816fc214d30ac87d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 14:46:30.735075 2013 raft_consensus.cc:348] T a454436a865e42a8b2fdeaccdb2b28f9 P 3a138c74c83a4052816fc214d30ac87d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5d18da021245748c1655fe7fed35b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46285 } } peers { permanent_uuid: "e4742b727bc24e89ad40347bfc4a8942" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38555 } } peers { permanent_uuid: "3a138c74c83a4052816fc214d30ac87d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41623 } }
I20260619 14:46:30.735198 2013 raft_consensus.cc:374] T a454436a865e42a8b2fdeaccdb2b28f9 P 3a138c74c83a4052816fc214d30ac87d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:46:30.735234 2013 raft_consensus.cc:729] T a454436a865e42a8b2fdeaccdb2b28f9 P 3a138c74c83a4052816fc214d30ac87d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a138c74c83a4052816fc214d30ac87d, State: Initialized, Role: FOLLOWER
I20260619 14:46:30.735351 2013 consensus_queue.cc:260] T a454436a865e42a8b2fdeaccdb2b28f9 P 3a138c74c83a4052816fc214d30ac87d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5d18da021245748c1655fe7fed35b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46285 } } peers { permanent_uuid: "e4742b727bc24e89ad40347bfc4a8942" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38555 } } peers { permanent_uuid: "3a138c74c83a4052816fc214d30ac87d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41623 } }
I20260619 14:46:30.735620 2013 ts_tablet_manager.cc:1428] T a454436a865e42a8b2fdeaccdb2b28f9 P 3a138c74c83a4052816fc214d30ac87d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:46:30.735904 2039 ts_tablet_manager.cc:1918] T a40a76e4292f439d98589fce62ce5a33 P 5e5d18da021245748c1655fe7fed35b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:46:30.736066 2039 log.cc:1196] T a40a76e4292f439d98589fce62ce5a33 P 5e5d18da021245748c1655fe7fed35b1: Deleting WAL directory at /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880383132139-392-0/minicluster-data/ts-0/wal/wals/a40a76e4292f439d98589fce62ce5a33
I20260619 14:46:30.736428 2039 ts_tablet_manager.cc:1939] T a40a76e4292f439d98589fce62ce5a33 P 5e5d18da021245748c1655fe7fed35b1: Deleting consensus metadata
I20260619 14:46:30.736424 1493 catalog_manager.cc:4841] TS 3a138c74c83a4052816fc214d30ac87d (127.0.98.3:41623): tablet a40a76e4292f439d98589fce62ce5a33 (table test-table [id=d3603b8ea7ed49c49ac06d97802455c6]) successfully deleted
I20260619 14:46:30.737799 1493 catalog_manager.cc:4841] TS 5e5d18da021245748c1655fe7fed35b1 (127.0.98.1:46285): tablet a40a76e4292f439d98589fce62ce5a33 (table test-table [id=d3603b8ea7ed49c49ac06d97802455c6]) successfully deleted
W20260619 14:46:30.771282 1711 tablet.cc:2369] T a454436a865e42a8b2fdeaccdb2b28f9 P 5e5d18da021245748c1655fe7fed35b1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 14:46:30.777364 1991 tablet.cc:2369] T a454436a865e42a8b2fdeaccdb2b28f9 P 3a138c74c83a4052816fc214d30ac87d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 14:46:30.791034 1851 tablet.cc:2369] T a454436a865e42a8b2fdeaccdb2b28f9 P e4742b727bc24e89ad40347bfc4a8942: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:46:30.945755 2019 raft_consensus.cc:482] T a454436a865e42a8b2fdeaccdb2b28f9 P 3a138c74c83a4052816fc214d30ac87d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:46:30.945912 2019 raft_consensus.cc:504] T a454436a865e42a8b2fdeaccdb2b28f9 P 3a138c74c83a4052816fc214d30ac87d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5d18da021245748c1655fe7fed35b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46285 } } peers { permanent_uuid: "e4742b727bc24e89ad40347bfc4a8942" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38555 } } peers { permanent_uuid: "3a138c74c83a4052816fc214d30ac87d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41623 } }
I20260619 14:46:30.947113 2019 leader_election.cc:288] T a454436a865e42a8b2fdeaccdb2b28f9 P 3a138c74c83a4052816fc214d30ac87d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e5d18da021245748c1655fe7fed35b1 (127.0.98.1:46285), e4742b727bc24e89ad40347bfc4a8942 (127.0.98.2:38555)
I20260619 14:46:30.951854 1883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:30.947585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46285 (local address 127.0.98.3:39567)
0619 14:46:30.947828 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:30.947844 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:30.947954 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:30.948285 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:30.948288 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:30.948307 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:46:30.948455 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:30.948463 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:30.949360 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:30.949364 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:30.950416 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:30.950426 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:30.951311 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:30.951316 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:30.951422 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:30.951447 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 14:46:30.951538 (+ 91us) client_negotiation.cc:240] Negotiation successful
0619 14:46:30.951618 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158}
I20260619 14:46:30.951843 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:30.947790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39567 (local address 127.0.98.1:46285)
0619 14:46:30.947919 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:30.947923 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:30.947941 (+ 18us) server_negotiation.cc:407] Connection header received
0619 14:46:30.948079 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:30.948085 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:30.948145 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:30.948225 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:46:30.948578 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:30.949229 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:30.950579 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:30.951164 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:30.951267 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:30.951327 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:30.951565 (+ 238us) server_negotiation.cc:299] Negotiation successful
0619 14:46:30.951630 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 14:46:30.952256 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:30.947750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34031 (local address 127.0.98.2:38555)
0619 14:46:30.947886 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:30.947891 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:30.948148 (+ 257us) server_negotiation.cc:407] Connection header received
0619 14:46:30.948288 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:30.948291 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:30.948358 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:30.948457 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:46:30.948799 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:30.949601 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:30.950921 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:30.951594 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:30.951696 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:30.951751 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:30.952023 (+ 272us) server_negotiation.cc:299] Negotiation successful
0619 14:46:30.952122 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 14:46:30.952610 2043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:30.947633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38555 (local address 127.0.98.3:34031)
0619 14:46:30.948029 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:30.948045 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:30.948159 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:30.948473 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:30.948475 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:30.948489 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:46:30.948629 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:30.948636 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:30.949749 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:30.949752 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:30.950773 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:30.950783 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:30.951748 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:30.951753 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:30.951841 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:30.951875 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 14:46:30.952401 (+ 526us) client_negotiation.cc:240] Negotiation successful
0619 14:46:30.952486 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"mutex_wait_us":1,"thread_start_us":75,"threads_started":1}
I20260619 14:46:30.952745 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a454436a865e42a8b2fdeaccdb2b28f9" candidate_uuid: "3a138c74c83a4052816fc214d30ac87d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e5d18da021245748c1655fe7fed35b1" is_pre_election: true
I20260619 14:46:30.953042 1664 raft_consensus.cc:2454] T a454436a865e42a8b2fdeaccdb2b28f9 P 5e5d18da021245748c1655fe7fed35b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a138c74c83a4052816fc214d30ac87d in term 0.
I20260619 14:46:30.953208 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a454436a865e42a8b2fdeaccdb2b28f9" candidate_uuid: "3a138c74c83a4052816fc214d30ac87d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4742b727bc24e89ad40347bfc4a8942" is_pre_election: true
I20260619 14:46:30.953368 1804 raft_consensus.cc:2454] T a454436a865e42a8b2fdeaccdb2b28f9 P e4742b727bc24e89ad40347bfc4a8942 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a138c74c83a4052816fc214d30ac87d in term 0.
I20260619 14:46:30.953523 1877 leader_election.cc:302] T a454436a865e42a8b2fdeaccdb2b28f9 P 3a138c74c83a4052816fc214d30ac87d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a138c74c83a4052816fc214d30ac87d, 5e5d18da021245748c1655fe7fed35b1; no voters:
I20260619 14:46:30.953936 2019 raft_consensus.cc:2781] T a454436a865e42a8b2fdeaccdb2b28f9 P 3a138c74c83a4052816fc214d30ac87d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:46:30.954006 2019 raft_consensus.cc:482] T a454436a865e42a8b2fdeaccdb2b28f9 P 3a138c74c83a4052816fc214d30ac87d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:46:30.954044 2019 raft_consensus.cc:3037] T a454436a865e42a8b2fdeaccdb2b28f9 P 3a138c74c83a4052816fc214d30ac87d [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:46:30.955020 2019 raft_consensus.cc:504] T a454436a865e42a8b2fdeaccdb2b28f9 P 3a138c74c83a4052816fc214d30ac87d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5d18da021245748c1655fe7fed35b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46285 } } peers { permanent_uuid: "e4742b727bc24e89ad40347bfc4a8942" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38555 } } peers { permanent_uuid: "3a138c74c83a4052816fc214d30ac87d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41623 } }
I20260619 14:46:30.955444 2019 leader_election.cc:288] T a454436a865e42a8b2fdeaccdb2b28f9 P 3a138c74c83a4052816fc214d30ac87d [CANDIDATE]: Term 1 election: Requested vote from peers 5e5d18da021245748c1655fe7fed35b1 (127.0.98.1:46285), e4742b727bc24e89ad40347bfc4a8942 (127.0.98.2:38555)
I20260619 14:46:30.955847 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a454436a865e42a8b2fdeaccdb2b28f9" candidate_uuid: "3a138c74c83a4052816fc214d30ac87d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e5d18da021245748c1655fe7fed35b1"
I20260619 14:46:30.955909 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a454436a865e42a8b2fdeaccdb2b28f9" candidate_uuid: "3a138c74c83a4052816fc214d30ac87d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4742b727bc24e89ad40347bfc4a8942"
I20260619 14:46:30.956017 1664 raft_consensus.cc:3037] T a454436a865e42a8b2fdeaccdb2b28f9 P 5e5d18da021245748c1655fe7fed35b1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:46:30.956043 1804 raft_consensus.cc:3037] T a454436a865e42a8b2fdeaccdb2b28f9 P e4742b727bc24e89ad40347bfc4a8942 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:46:30.957326 1664 raft_consensus.cc:2454] T a454436a865e42a8b2fdeaccdb2b28f9 P 5e5d18da021245748c1655fe7fed35b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a138c74c83a4052816fc214d30ac87d in term 1.
I20260619 14:46:30.957326 1804 raft_consensus.cc:2454] T a454436a865e42a8b2fdeaccdb2b28f9 P e4742b727bc24e89ad40347bfc4a8942 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a138c74c83a4052816fc214d30ac87d in term 1.
I20260619 14:46:30.957672 1877 leader_election.cc:302] T a454436a865e42a8b2fdeaccdb2b28f9 P 3a138c74c83a4052816fc214d30ac87d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a138c74c83a4052816fc214d30ac87d, 5e5d18da021245748c1655fe7fed35b1; no voters:
I20260619 14:46:30.958104 2019 raft_consensus.cc:2781] T a454436a865e42a8b2fdeaccdb2b28f9 P 3a138c74c83a4052816fc214d30ac87d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:46:30.958491 2019 raft_consensus.cc:686] T a454436a865e42a8b2fdeaccdb2b28f9 P 3a138c74c83a4052816fc214d30ac87d [term 1 LEADER]: Becoming Leader. State: Replica: 3a138c74c83a4052816fc214d30ac87d, State: Running, Role: LEADER
I20260619 14:46:30.958778 2019 consensus_queue.cc:237] T a454436a865e42a8b2fdeaccdb2b28f9 P 3a138c74c83a4052816fc214d30ac87d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5d18da021245748c1655fe7fed35b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46285 } } peers { permanent_uuid: "e4742b727bc24e89ad40347bfc4a8942" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38555 } } peers { permanent_uuid: "3a138c74c83a4052816fc214d30ac87d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41623 } }
I20260619 14:46:30.961617 1506 catalog_manager.cc:5495] T a454436a865e42a8b2fdeaccdb2b28f9 P 3a138c74c83a4052816fc214d30ac87d reported cstate change: term changed from 0 to 1, leader changed from <none> to 3a138c74c83a4052816fc214d30ac87d (127.0.98.3). New cstate: current_term: 1 leader_uuid: "3a138c74c83a4052816fc214d30ac87d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5d18da021245748c1655fe7fed35b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4742b727bc24e89ad40347bfc4a8942" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38555 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a138c74c83a4052816fc214d30ac87d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41623 } health_report { overall_health: HEALTHY } } }
Jun 19 14:46:31 dist-test-slave-2sb1 krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880389, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:46:31.008325 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:30.998860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58248 (local address 127.0.98.62:46827)
0619 14:46:30.998993 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:30.998997 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:30.999329 (+ 332us) server_negotiation.cc:407] Connection header received
0619 14:46:30.999377 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:30.999379 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:30.999416 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:30.999504 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:31.000217 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:31.000803 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:31.001587 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:31.001803 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:31.001914 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:31.004586 (+ 2672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:31.004597 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:31.004599 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:31.004622 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:31.006472 (+ 1850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:31.006999 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:31.007011 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:31.007013 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:31.007090 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:31.007358 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:31.007365 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:31.007367 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:31.007557 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:31.007659 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:31.007967 (+ 308us) server_negotiation.cc:299] Negotiation successful
0619 14:46:31.008120 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":47}
I20260619 14:46:31.008347 2051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:30.998789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46827 (local address 127.0.0.1:58248)
0619 14:46:30.999099 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:30.999110 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:30.999270 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:30.999522 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:30.999525 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:30.999967 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:46:31.000081 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:31.000088 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:31.000909 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:31.000915 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:31.001469 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:31.001477 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:31.001920 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:31.001923 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:31.002029 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:31.002673 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:46:31.002686 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:46:31.004400 (+ 1714us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:46:31.006634 (+ 2234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:31.006645 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:31.006652 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:31.006898 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:31.007187 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:31.007190 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:31.007192 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:31.007271 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:31.007673 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:46:31.007683 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:46:31.007794 (+ 111us) client_negotiation.cc:769] Sending connection context
0619 14:46:31.007912 (+ 118us) client_negotiation.cc:240] Negotiation successful
0619 14:46:31.008183 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"mutex_wait_us":102,"thread_start_us":114,"threads_started":1}
I20260619 14:46:31.013330 1506 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58248:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 14:46:31.014180 1506 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:46:31.020147 1644 tablet_service.cc:1467] Processing CreateTablet for tablet 3b4f4452968d4dde928b693bab04004b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0e043745bfaf404fb5d90c71ef6cc0ba]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 14:46:31.020298 1784 tablet_service.cc:1467] Processing CreateTablet for tablet 3b4f4452968d4dde928b693bab04004b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0e043745bfaf404fb5d90c71ef6cc0ba]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 14:46:31.020560 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b4f4452968d4dde928b693bab04004b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:46:31.020885 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b4f4452968d4dde928b693bab04004b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:46:31.023641 2014 tablet_bootstrap.cc:492] T 3b4f4452968d4dde928b693bab04004b P 5e5d18da021245748c1655fe7fed35b1: Bootstrap starting.
I20260619 14:46:31.024490 2015 tablet_bootstrap.cc:492] T 3b4f4452968d4dde928b693bab04004b P e4742b727bc24e89ad40347bfc4a8942: Bootstrap starting.
I20260619 14:46:31.024734 2014 tablet_bootstrap.cc:654] T 3b4f4452968d4dde928b693bab04004b P 5e5d18da021245748c1655fe7fed35b1: Neither blocks nor log segments found. Creating new log.
I20260619 14:46:31.024749 1924 tablet_service.cc:1467] Processing CreateTablet for tablet 3b4f4452968d4dde928b693bab04004b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0e043745bfaf404fb5d90c71ef6cc0ba]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 14:46:31.025133 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b4f4452968d4dde928b693bab04004b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:46:31.025621 2015 tablet_bootstrap.cc:654] T 3b4f4452968d4dde928b693bab04004b P e4742b727bc24e89ad40347bfc4a8942: Neither blocks nor log segments found. Creating new log.
I20260619 14:46:31.026051 2014 tablet_bootstrap.cc:492] T 3b4f4452968d4dde928b693bab04004b P 5e5d18da021245748c1655fe7fed35b1: No bootstrap required, opened a new log
I20260619 14:46:31.026144 2014 ts_tablet_manager.cc:1397] T 3b4f4452968d4dde928b693bab04004b P 5e5d18da021245748c1655fe7fed35b1: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 14:46:31.026612 2014 raft_consensus.cc:348] T 3b4f4452968d4dde928b693bab04004b P 5e5d18da021245748c1655fe7fed35b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5d18da021245748c1655fe7fed35b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46285 } } peers { permanent_uuid: "e4742b727bc24e89ad40347bfc4a8942" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38555 } } peers { permanent_uuid: "3a138c74c83a4052816fc214d30ac87d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41623 } }
I20260619 14:46:31.026758 2014 raft_consensus.cc:374] T 3b4f4452968d4dde928b693bab04004b P 5e5d18da021245748c1655fe7fed35b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:46:31.026809 2014 raft_consensus.cc:729] T 3b4f4452968d4dde928b693bab04004b P 5e5d18da021245748c1655fe7fed35b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e5d18da021245748c1655fe7fed35b1, State: Initialized, Role: FOLLOWER
I20260619 14:46:31.026978 2014 consensus_queue.cc:260] T 3b4f4452968d4dde928b693bab04004b P 5e5d18da021245748c1655fe7fed35b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5d18da021245748c1655fe7fed35b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46285 } } peers { permanent_uuid: "e4742b727bc24e89ad40347bfc4a8942" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38555 } } peers { permanent_uuid: "3a138c74c83a4052816fc214d30ac87d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41623 } }
I20260619 14:46:31.027347 2014 ts_tablet_manager.cc:1428] T 3b4f4452968d4dde928b693bab04004b P 5e5d18da021245748c1655fe7fed35b1: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260619 14:46:31.027664 2021 tablet_replica.cc:439] T 3b4f4452968d4dde928b693bab04004b P 5e5d18da021245748c1655fe7fed35b1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5d18da021245748c1655fe7fed35b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46285 } } peers { permanent_uuid: "e4742b727bc24e89ad40347bfc4a8942" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38555 } } peers { permanent_uuid: "3a138c74c83a4052816fc214d30ac87d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41623 } } }
I20260619 14:46:31.027801 2021 tablet_replica.cc:442] T 3b4f4452968d4dde928b693bab04004b P 5e5d18da021245748c1655fe7fed35b1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:46:31.028297 2013 tablet_bootstrap.cc:492] T 3b4f4452968d4dde928b693bab04004b P 3a138c74c83a4052816fc214d30ac87d: Bootstrap starting.
I20260619 14:46:31.029537 2013 tablet_bootstrap.cc:654] T 3b4f4452968d4dde928b693bab04004b P 3a138c74c83a4052816fc214d30ac87d: Neither blocks nor log segments found. Creating new log.
I20260619 14:46:31.030398 2015 tablet_bootstrap.cc:492] T 3b4f4452968d4dde928b693bab04004b P e4742b727bc24e89ad40347bfc4a8942: No bootstrap required, opened a new log
I20260619 14:46:31.030537 2015 ts_tablet_manager.cc:1397] T 3b4f4452968d4dde928b693bab04004b P e4742b727bc24e89ad40347bfc4a8942: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260619 14:46:31.030939 2013 tablet_bootstrap.cc:492] T 3b4f4452968d4dde928b693bab04004b P 3a138c74c83a4052816fc214d30ac87d: No bootstrap required, opened a new log
I20260619 14:46:31.031036 2013 ts_tablet_manager.cc:1397] T 3b4f4452968d4dde928b693bab04004b P 3a138c74c83a4052816fc214d30ac87d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 14:46:31.031070 2015 raft_consensus.cc:348] T 3b4f4452968d4dde928b693bab04004b P e4742b727bc24e89ad40347bfc4a8942 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5d18da021245748c1655fe7fed35b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46285 } } peers { permanent_uuid: "e4742b727bc24e89ad40347bfc4a8942" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38555 } } peers { permanent_uuid: "3a138c74c83a4052816fc214d30ac87d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41623 } }
I20260619 14:46:31.031203 2015 raft_consensus.cc:374] T 3b4f4452968d4dde928b693bab04004b P e4742b727bc24e89ad40347bfc4a8942 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:46:31.031244 2015 raft_consensus.cc:729] T 3b4f4452968d4dde928b693bab04004b P e4742b727bc24e89ad40347bfc4a8942 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4742b727bc24e89ad40347bfc4a8942, State: Initialized, Role: FOLLOWER
I20260619 14:46:31.031405 2015 consensus_queue.cc:260] T 3b4f4452968d4dde928b693bab04004b P e4742b727bc24e89ad40347bfc4a8942 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5d18da021245748c1655fe7fed35b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46285 } } peers { permanent_uuid: "e4742b727bc24e89ad40347bfc4a8942" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38555 } } peers { permanent_uuid: "3a138c74c83a4052816fc214d30ac87d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41623 } }
I20260619 14:46:31.031917 2015 ts_tablet_manager.cc:1428] T 3b4f4452968d4dde928b693bab04004b P e4742b727bc24e89ad40347bfc4a8942: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:46:31.031925 2013 raft_consensus.cc:348] T 3b4f4452968d4dde928b693bab04004b P 3a138c74c83a4052816fc214d30ac87d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5d18da021245748c1655fe7fed35b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46285 } } peers { permanent_uuid: "e4742b727bc24e89ad40347bfc4a8942" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38555 } } peers { permanent_uuid: "3a138c74c83a4052816fc214d30ac87d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41623 } }
I20260619 14:46:31.032087 2013 raft_consensus.cc:374] T 3b4f4452968d4dde928b693bab04004b P 3a138c74c83a4052816fc214d30ac87d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:46:31.032130 2013 raft_consensus.cc:729] T 3b4f4452968d4dde928b693bab04004b P 3a138c74c83a4052816fc214d30ac87d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a138c74c83a4052816fc214d30ac87d, State: Initialized, Role: FOLLOWER
I20260619 14:46:31.032276 2013 consensus_queue.cc:260] T 3b4f4452968d4dde928b693bab04004b P 3a138c74c83a4052816fc214d30ac87d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5d18da021245748c1655fe7fed35b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46285 } } peers { permanent_uuid: "e4742b727bc24e89ad40347bfc4a8942" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38555 } } peers { permanent_uuid: "3a138c74c83a4052816fc214d30ac87d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41623 } }
I20260619 14:46:31.032681 2013 ts_tablet_manager.cc:1428] T 3b4f4452968d4dde928b693bab04004b P 3a138c74c83a4052816fc214d30ac87d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 14:46:31.032951 2020 tablet_replica.cc:439] T 3b4f4452968d4dde928b693bab04004b P e4742b727bc24e89ad40347bfc4a8942: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5d18da021245748c1655fe7fed35b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46285 } } peers { permanent_uuid: "e4742b727bc24e89ad40347bfc4a8942" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38555 } } peers { permanent_uuid: "3a138c74c83a4052816fc214d30ac87d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41623 } } }
I20260619 14:46:31.033164 2020 tablet_replica.cc:442] T 3b4f4452968d4dde928b693bab04004b P e4742b727bc24e89ad40347bfc4a8942: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:46:31.033442 2019 tablet_replica.cc:439] T 3b4f4452968d4dde928b693bab04004b P 3a138c74c83a4052816fc214d30ac87d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5d18da021245748c1655fe7fed35b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46285 } } peers { permanent_uuid: "e4742b727bc24e89ad40347bfc4a8942" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38555 } } peers { permanent_uuid: "3a138c74c83a4052816fc214d30ac87d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41623 } } }
I20260619 14:46:31.033607 2019 tablet_replica.cc:442] T 3b4f4452968d4dde928b693bab04004b P 3a138c74c83a4052816fc214d30ac87d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:46:31.123155 2020 raft_consensus.cc:482] T 3b4f4452968d4dde928b693bab04004b P e4742b727bc24e89ad40347bfc4a8942 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:46:31.123318 2020 raft_consensus.cc:504] T 3b4f4452968d4dde928b693bab04004b P e4742b727bc24e89ad40347bfc4a8942 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5d18da021245748c1655fe7fed35b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46285 } } peers { permanent_uuid: "e4742b727bc24e89ad40347bfc4a8942" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38555 } } peers { permanent_uuid: "3a138c74c83a4052816fc214d30ac87d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41623 } }
I20260619 14:46:31.124787 2020 leader_election.cc:288] T 3b4f4452968d4dde928b693bab04004b P e4742b727bc24e89ad40347bfc4a8942 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e5d18da021245748c1655fe7fed35b1 (127.0.98.1:46285), 3a138c74c83a4052816fc214d30ac87d (127.0.98.3:41623)
I20260619 14:46:31.126895 2019 raft_consensus.cc:482] T 3b4f4452968d4dde928b693bab04004b P 3a138c74c83a4052816fc214d30ac87d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:46:31.127048 2019 raft_consensus.cc:504] T 3b4f4452968d4dde928b693bab04004b P 3a138c74c83a4052816fc214d30ac87d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5d18da021245748c1655fe7fed35b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46285 } } peers { permanent_uuid: "e4742b727bc24e89ad40347bfc4a8942" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38555 } } peers { permanent_uuid: "3a138c74c83a4052816fc214d30ac87d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41623 } }
I20260619 14:46:31.127475 2019 leader_election.cc:288] T 3b4f4452968d4dde928b693bab04004b P 3a138c74c83a4052816fc214d30ac87d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e5d18da021245748c1655fe7fed35b1 (127.0.98.1:46285), e4742b727bc24e89ad40347bfc4a8942 (127.0.98.2:38555)
I20260619 14:46:31.127925 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b4f4452968d4dde928b693bab04004b" candidate_uuid: "3a138c74c83a4052816fc214d30ac87d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e5d18da021245748c1655fe7fed35b1" is_pre_election: true
I20260619 14:46:31.128122 1664 raft_consensus.cc:2454] T 3b4f4452968d4dde928b693bab04004b P 5e5d18da021245748c1655fe7fed35b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a138c74c83a4052816fc214d30ac87d in term 0.
I20260619 14:46:31.128114 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b4f4452968d4dde928b693bab04004b" candidate_uuid: "3a138c74c83a4052816fc214d30ac87d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4742b727bc24e89ad40347bfc4a8942" is_pre_election: true
I20260619 14:46:31.128276 1804 raft_consensus.cc:2454] T 3b4f4452968d4dde928b693bab04004b P e4742b727bc24e89ad40347bfc4a8942 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3a138c74c83a4052816fc214d30ac87d in term 0.
I20260619 14:46:31.128480 1877 leader_election.cc:302] T 3b4f4452968d4dde928b693bab04004b P 3a138c74c83a4052816fc214d30ac87d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a138c74c83a4052816fc214d30ac87d, 5e5d18da021245748c1655fe7fed35b1; no voters:
I20260619 14:46:31.128783 2019 raft_consensus.cc:2781] T 3b4f4452968d4dde928b693bab04004b P 3a138c74c83a4052816fc214d30ac87d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:46:31.128847 2019 raft_consensus.cc:482] T 3b4f4452968d4dde928b693bab04004b P 3a138c74c83a4052816fc214d30ac87d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:46:31.128911 2019 raft_consensus.cc:3037] T 3b4f4452968d4dde928b693bab04004b P 3a138c74c83a4052816fc214d30ac87d [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:46:31.130297 2019 raft_consensus.cc:504] T 3b4f4452968d4dde928b693bab04004b P 3a138c74c83a4052816fc214d30ac87d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5d18da021245748c1655fe7fed35b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46285 } } peers { permanent_uuid: "e4742b727bc24e89ad40347bfc4a8942" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38555 } } peers { permanent_uuid: "3a138c74c83a4052816fc214d30ac87d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41623 } }
I20260619 14:46:31.130766 2019 leader_election.cc:288] T 3b4f4452968d4dde928b693bab04004b P 3a138c74c83a4052816fc214d30ac87d [CANDIDATE]: Term 1 election: Requested vote from peers 5e5d18da021245748c1655fe7fed35b1 (127.0.98.1:46285), e4742b727bc24e89ad40347bfc4a8942 (127.0.98.2:38555)
I20260619 14:46:31.131084 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:31.125799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38771 (local address 127.0.98.3:41623)
0619 14:46:31.125958 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:31.125962 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:31.125975 (+ 13us) server_negotiation.cc:407] Connection header received
0619 14:46:31.126201 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:31.126205 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:31.126278 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:31.126364 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:46:31.126756 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:31.127769 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:31.129764 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:31.130267 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:31.130338 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:31.130390 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:31.130826 (+ 436us) server_negotiation.cc:299] Negotiation successful
0619 14:46:31.130931 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 14:46:31.131124 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:31.125228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58693 (local address 127.0.98.1:46285)
0619 14:46:31.125419 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:31.125425 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:31.125574 (+ 149us) server_negotiation.cc:407] Connection header received
0619 14:46:31.125854 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:31.125860 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:31.125929 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:31.126024 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:46:31.126380 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:31.127343 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:31.129565 (+ 2222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:31.130278 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:31.130372 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:31.130447 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:31.130858 (+ 411us) server_negotiation.cc:299] Negotiation successful
0619 14:46:31.130961 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260619 14:46:31.131281 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b4f4452968d4dde928b693bab04004b" candidate_uuid: "3a138c74c83a4052816fc214d30ac87d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4742b727bc24e89ad40347bfc4a8942"
I20260619 14:46:31.131422 1804 raft_consensus.cc:3037] T 3b4f4452968d4dde928b693bab04004b P e4742b727bc24e89ad40347bfc4a8942 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:46:31.131546 2056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:31.125561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41623 (local address 127.0.98.2:38771)
0619 14:46:31.125836 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:31.125853 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:31.125944 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:31.126392 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:31.126395 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:31.126415 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:46:31.126535 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:31.126541 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:31.127911 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:31.127914 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:31.129570 (+ 1656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:31.129593 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:31.130530 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:31.130536 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:31.130644 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:31.130663 (+ 19us) client_negotiation.cc:769] Sending connection context
0619 14:46:31.131239 (+ 576us) client_negotiation.cc:240] Negotiation successful
0619 14:46:31.131344 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":62,"threads_started":1}
I20260619 14:46:31.131713 2055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:31.125142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46285 (local address 127.0.98.2:58693)
0619 14:46:31.125453 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:31.125469 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:31.125673 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:31.126053 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:31.126057 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:31.126078 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:46:31.126217 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:31.126226 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:31.127464 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:31.127471 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:31.129405 (+ 1934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:31.129417 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:31.130558 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:31.130561 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:31.130687 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:31.130716 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 14:46:31.131340 (+ 624us) client_negotiation.cc:240] Negotiation successful
0619 14:46:31.131489 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":99,"threads_started":1}
I20260619 14:46:31.132077 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b4f4452968d4dde928b693bab04004b" candidate_uuid: "3a138c74c83a4052816fc214d30ac87d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e5d18da021245748c1655fe7fed35b1"
I20260619 14:46:31.132215 1664 raft_consensus.cc:3037] T 3b4f4452968d4dde928b693bab04004b P 5e5d18da021245748c1655fe7fed35b1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:46:31.132180 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b4f4452968d4dde928b693bab04004b" candidate_uuid: "e4742b727bc24e89ad40347bfc4a8942" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a138c74c83a4052816fc214d30ac87d" is_pre_election: true
I20260619 14:46:31.132179 1663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b4f4452968d4dde928b693bab04004b" candidate_uuid: "e4742b727bc24e89ad40347bfc4a8942" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e5d18da021245748c1655fe7fed35b1" is_pre_election: true
I20260619 14:46:31.132365 1944 raft_consensus.cc:2379] T 3b4f4452968d4dde928b693bab04004b P 3a138c74c83a4052816fc214d30ac87d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e4742b727bc24e89ad40347bfc4a8942 in current term 1: Already voted for candidate 3a138c74c83a4052816fc214d30ac87d in this term.
I20260619 14:46:31.132781 1804 raft_consensus.cc:2454] T 3b4f4452968d4dde928b693bab04004b P e4742b727bc24e89ad40347bfc4a8942 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a138c74c83a4052816fc214d30ac87d in term 1.
I20260619 14:46:31.133155 1737 leader_election.cc:302] T 3b4f4452968d4dde928b693bab04004b P e4742b727bc24e89ad40347bfc4a8942 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4742b727bc24e89ad40347bfc4a8942; no voters: 3a138c74c83a4052816fc214d30ac87d, 5e5d18da021245748c1655fe7fed35b1
I20260619 14:46:31.133445 2020 raft_consensus.cc:2726] T 3b4f4452968d4dde928b693bab04004b P e4742b727bc24e89ad40347bfc4a8942 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260619 14:46:31.133658 1664 raft_consensus.cc:2454] T 3b4f4452968d4dde928b693bab04004b P 5e5d18da021245748c1655fe7fed35b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a138c74c83a4052816fc214d30ac87d in term 1.
I20260619 14:46:31.134249 1877 leader_election.cc:302] T 3b4f4452968d4dde928b693bab04004b P 3a138c74c83a4052816fc214d30ac87d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3a138c74c83a4052816fc214d30ac87d, 5e5d18da021245748c1655fe7fed35b1, e4742b727bc24e89ad40347bfc4a8942; no voters:
I20260619 14:46:31.134585 2019 raft_consensus.cc:2781] T 3b4f4452968d4dde928b693bab04004b P 3a138c74c83a4052816fc214d30ac87d [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:46:31.134689 2019 raft_consensus.cc:686] T 3b4f4452968d4dde928b693bab04004b P 3a138c74c83a4052816fc214d30ac87d [term 1 LEADER]: Becoming Leader. State: Replica: 3a138c74c83a4052816fc214d30ac87d, State: Running, Role: LEADER
I20260619 14:46:31.134838 2019 consensus_queue.cc:237] T 3b4f4452968d4dde928b693bab04004b P 3a138c74c83a4052816fc214d30ac87d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5d18da021245748c1655fe7fed35b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46285 } } peers { permanent_uuid: "e4742b727bc24e89ad40347bfc4a8942" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38555 } } peers { permanent_uuid: "3a138c74c83a4052816fc214d30ac87d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41623 } }
I20260619 14:46:31.135659 2044 tablet_replica.cc:439] T 3b4f4452968d4dde928b693bab04004b P 3a138c74c83a4052816fc214d30ac87d: TxnStatusTablet state changed. Reason: New leader 3a138c74c83a4052816fc214d30ac87d. Latest consensus state: current_term: 1 leader_uuid: "3a138c74c83a4052816fc214d30ac87d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5d18da021245748c1655fe7fed35b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46285 } } peers { permanent_uuid: "e4742b727bc24e89ad40347bfc4a8942" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38555 } } peers { permanent_uuid: "3a138c74c83a4052816fc214d30ac87d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41623 } } }
I20260619 14:46:31.135783 2044 tablet_replica.cc:442] T 3b4f4452968d4dde928b693bab04004b P 3a138c74c83a4052816fc214d30ac87d: This TxnStatusTablet replica's current role is: LEADER
I20260619 14:46:31.136129 2058 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 14:46:31.136359 2058 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 14:46:31.136955 1506 catalog_manager.cc:5495] T 3b4f4452968d4dde928b693bab04004b P 3a138c74c83a4052816fc214d30ac87d reported cstate change: term changed from 0 to 1, leader changed from <none> to 3a138c74c83a4052816fc214d30ac87d (127.0.98.3). New cstate: current_term: 1 leader_uuid: "3a138c74c83a4052816fc214d30ac87d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5d18da021245748c1655fe7fed35b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4742b727bc24e89ad40347bfc4a8942" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38555 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3a138c74c83a4052816fc214d30ac87d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41623 } health_report { overall_health: HEALTHY } } }
I20260619 14:46:31.164603 2051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:31.160041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41623 (local address 127.0.0.1:40858)
0619 14:46:31.160292 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:31.160307 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:31.160423 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:31.160822 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:31.160825 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:31.160834 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 14:46:31.160950 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:31.160957 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:31.161638 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:31.161641 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:31.162652 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:31.162665 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:31.163225 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:31.163229 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:31.163325 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:31.163387 (+ 62us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 14:46:31.164248 (+ 861us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 14:46:31.164266 (+ 18us) client_negotiation.cc:769] Sending connection context
0619 14:46:31.164347 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 14:46:31.164451 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152}
I20260619 14:46:31.164602 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:31.160140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40858 (local address 127.0.98.3:41623)
0619 14:46:31.160272 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:31.160277 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:31.160363 (+ 86us) server_negotiation.cc:407] Connection header received
0619 14:46:31.160556 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:31.160559 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:31.160611 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:31.160789 (+ 178us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:46:31.161040 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:31.161523 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:31.162857 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:31.163095 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:31.163295 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:31.163582 (+ 287us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:46:31.164128 (+ 546us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:46:31.164235 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:31.164405 (+ 170us) server_negotiation.cc:299] Negotiation successful
0619 14:46:31.164462 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 14:46:31.168183 1804 raft_consensus.cc:1261] T 3b4f4452968d4dde928b693bab04004b P e4742b727bc24e89ad40347bfc4a8942 [term 1 FOLLOWER]: Refusing update from remote peer 3a138c74c83a4052816fc214d30ac87d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:46:31.168241 1664 raft_consensus.cc:1261] T 3b4f4452968d4dde928b693bab04004b P 5e5d18da021245748c1655fe7fed35b1 [term 1 FOLLOWER]: Refusing update from remote peer 3a138c74c83a4052816fc214d30ac87d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:46:31.168779 2019 consensus_queue.cc:1035] T 3b4f4452968d4dde928b693bab04004b P 3a138c74c83a4052816fc214d30ac87d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e5d18da021245748c1655fe7fed35b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:46:31.168982 2044 consensus_queue.cc:1035] T 3b4f4452968d4dde928b693bab04004b P 3a138c74c83a4052816fc214d30ac87d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4742b727bc24e89ad40347bfc4a8942" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:46:31.170842 2021 tablet_replica.cc:439] T 3b4f4452968d4dde928b693bab04004b P 5e5d18da021245748c1655fe7fed35b1: TxnStatusTablet state changed. Reason: New leader 3a138c74c83a4052816fc214d30ac87d. Latest consensus state: current_term: 1 leader_uuid: "3a138c74c83a4052816fc214d30ac87d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5d18da021245748c1655fe7fed35b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46285 } } peers { permanent_uuid: "e4742b727bc24e89ad40347bfc4a8942" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38555 } } peers { permanent_uuid: "3a138c74c83a4052816fc214d30ac87d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41623 } } }
I20260619 14:46:31.170986 2021 tablet_replica.cc:442] T 3b4f4452968d4dde928b693bab04004b P 5e5d18da021245748c1655fe7fed35b1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:46:31.172394 2019 tablet_replica.cc:439] T 3b4f4452968d4dde928b693bab04004b P 3a138c74c83a4052816fc214d30ac87d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3a138c74c83a4052816fc214d30ac87d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5d18da021245748c1655fe7fed35b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46285 } } peers { permanent_uuid: "e4742b727bc24e89ad40347bfc4a8942" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38555 } } peers { permanent_uuid: "3a138c74c83a4052816fc214d30ac87d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41623 } } }
I20260619 14:46:31.172555 2019 tablet_replica.cc:442] T 3b4f4452968d4dde928b693bab04004b P 3a138c74c83a4052816fc214d30ac87d: This TxnStatusTablet replica's current role is: LEADER
I20260619 14:46:31.172627 2020 tablet_replica.cc:439] T 3b4f4452968d4dde928b693bab04004b P e4742b727bc24e89ad40347bfc4a8942: TxnStatusTablet state changed. Reason: New leader 3a138c74c83a4052816fc214d30ac87d. Latest consensus state: current_term: 1 leader_uuid: "3a138c74c83a4052816fc214d30ac87d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5d18da021245748c1655fe7fed35b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46285 } } peers { permanent_uuid: "e4742b727bc24e89ad40347bfc4a8942" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38555 } } peers { permanent_uuid: "3a138c74c83a4052816fc214d30ac87d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41623 } } }
I20260619 14:46:31.172789 2020 tablet_replica.cc:442] T 3b4f4452968d4dde928b693bab04004b P e4742b727bc24e89ad40347bfc4a8942: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:46:31.175619 2020 tablet_replica.cc:439] T 3b4f4452968d4dde928b693bab04004b P e4742b727bc24e89ad40347bfc4a8942: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3a138c74c83a4052816fc214d30ac87d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5d18da021245748c1655fe7fed35b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46285 } } peers { permanent_uuid: "e4742b727bc24e89ad40347bfc4a8942" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38555 } } peers { permanent_uuid: "3a138c74c83a4052816fc214d30ac87d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41623 } } }
I20260619 14:46:31.175771 2020 tablet_replica.cc:442] T 3b4f4452968d4dde928b693bab04004b P e4742b727bc24e89ad40347bfc4a8942: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:46:31.177737 2044 tablet_replica.cc:439] T 3b4f4452968d4dde928b693bab04004b P 3a138c74c83a4052816fc214d30ac87d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3a138c74c83a4052816fc214d30ac87d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5d18da021245748c1655fe7fed35b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46285 } } peers { permanent_uuid: "e4742b727bc24e89ad40347bfc4a8942" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38555 } } peers { permanent_uuid: "3a138c74c83a4052816fc214d30ac87d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41623 } } }
I20260619 14:46:31.177872 2044 tablet_replica.cc:442] T 3b4f4452968d4dde928b693bab04004b P 3a138c74c83a4052816fc214d30ac87d: This TxnStatusTablet replica's current role is: LEADER
I20260619 14:46:31.179565 2027 mvcc.cc:205] Tried to move back new op lower bound from 7298582082218274816 to 7298582082089816064. Current Snapshot: MvccSnapshot[applied={T|T < 7298582082218274816}]
I20260619 14:46:31.180047 2021 tablet_replica.cc:439] T 3b4f4452968d4dde928b693bab04004b P 5e5d18da021245748c1655fe7fed35b1: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3a138c74c83a4052816fc214d30ac87d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5d18da021245748c1655fe7fed35b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46285 } } peers { permanent_uuid: "e4742b727bc24e89ad40347bfc4a8942" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38555 } } peers { permanent_uuid: "3a138c74c83a4052816fc214d30ac87d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41623 } } }
I20260619 14:46:31.180177 2021 tablet_replica.cc:442] T 3b4f4452968d4dde928b693bab04004b P 5e5d18da021245748c1655fe7fed35b1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:46:31.187101 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:31.182321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40872 (local address 127.0.98.3:41623)
0619 14:46:31.182437 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:31.182442 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:31.183383 (+ 941us) server_negotiation.cc:407] Connection header received
0619 14:46:31.183487 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:31.183490 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:31.183545 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:31.183630 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:46:31.183997 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:31.184798 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:31.185775 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:31.186131 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:31.186236 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:31.186632 (+ 396us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:46:31.186717 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:46:31.186791 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:31.186889 (+ 98us) server_negotiation.cc:299] Negotiation successful
0619 14:46:31.186943 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 14:46:31.201419 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:31.197218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50505 (local address 127.0.98.3:41623)
0619 14:46:31.197367 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:31.197372 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:31.197644 (+ 272us) server_negotiation.cc:407] Connection header received
0619 14:46:31.197905 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:31.197909 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:31.197966 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:31.198081 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:46:31.198411 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:31.199045 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:31.200017 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:31.200270 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:31.200401 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:31.200796 (+ 395us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:46:31.200872 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:46:31.200963 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:31.201087 (+ 124us) server_negotiation.cc:299] Negotiation successful
0619 14:46:31.201202 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":75,"server-negotiator.queue_time_us":52}
I20260619 14:46:31.201426 2068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:31.197173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41623 (local address 127.0.98.3:50505)
0619 14:46:31.197528 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:31.197543 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:31.197739 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:31.198089 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:31.198091 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:31.198109 (+ 18us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 14:46:31.198239 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:31.198246 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:31.199164 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:31.199169 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:31.199892 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:31.199900 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:31.200412 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:31.200416 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:31.200545 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:31.200593 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 14:46:31.200968 (+ 375us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 14:46:31.200977 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 14:46:31.201076 (+ 99us) client_negotiation.cc:240] Negotiation successful
0619 14:46:31.201220 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"mutex_wait_us":33,"thread_start_us":153,"threads_started":1}
I20260619 14:46:31.209797 1804 raft_consensus.cc:1261] T a454436a865e42a8b2fdeaccdb2b28f9 P e4742b727bc24e89ad40347bfc4a8942 [term 1 FOLLOWER]: Refusing update from remote peer 3a138c74c83a4052816fc214d30ac87d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:46:31.209829 1664 raft_consensus.cc:1261] T a454436a865e42a8b2fdeaccdb2b28f9 P 5e5d18da021245748c1655fe7fed35b1 [term 1 FOLLOWER]: Refusing update from remote peer 3a138c74c83a4052816fc214d30ac87d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:46:31.210253 2063 consensus_queue.cc:1035] T a454436a865e42a8b2fdeaccdb2b28f9 P 3a138c74c83a4052816fc214d30ac87d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e5d18da021245748c1655fe7fed35b1" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:46:31.210546 2044 consensus_queue.cc:1035] T a454436a865e42a8b2fdeaccdb2b28f9 P 3a138c74c83a4052816fc214d30ac87d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4742b727bc24e89ad40347bfc4a8942" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:46:31.259542 1507 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58246:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:46:31.259786 1507 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58246:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:46:31.262295 1507 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 11ee01cd7a424422a7e03f596fd13d49: Sending DeleteTablet for 3 replicas of tablet a454436a865e42a8b2fdeaccdb2b28f9
I20260619 14:46:31.263149 1644 tablet_service.cc:1514] Processing DeleteTablet for tablet a454436a865e42a8b2fdeaccdb2b28f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:46:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33672
I20260619 14:46:31.263640 1924 tablet_service.cc:1514] Processing DeleteTablet for tablet a454436a865e42a8b2fdeaccdb2b28f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:46:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40842
I20260619 14:46:31.263998 2074 tablet_replica.cc:330] T a454436a865e42a8b2fdeaccdb2b28f9 P 5e5d18da021245748c1655fe7fed35b1: stopping tablet replica
I20260619 14:46:31.264160 2074 raft_consensus.cc:2229] T a454436a865e42a8b2fdeaccdb2b28f9 P 5e5d18da021245748c1655fe7fed35b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:46:31.264179 1784 tablet_service.cc:1514] Processing DeleteTablet for tablet a454436a865e42a8b2fdeaccdb2b28f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:46:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46190
I20260619 14:46:31.264307 2074 raft_consensus.cc:2258] T a454436a865e42a8b2fdeaccdb2b28f9 P 5e5d18da021245748c1655fe7fed35b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:46:31.265168 2074 ts_tablet_manager.cc:1905] T a454436a865e42a8b2fdeaccdb2b28f9 P 5e5d18da021245748c1655fe7fed35b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:46:31.266865 2075 tablet_replica.cc:330] T a454436a865e42a8b2fdeaccdb2b28f9 P 3a138c74c83a4052816fc214d30ac87d: stopping tablet replica
I20260619 14:46:31.267127 2075 raft_consensus.cc:2229] T a454436a865e42a8b2fdeaccdb2b28f9 P 3a138c74c83a4052816fc214d30ac87d [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:46:31.267576 2076 tablet_replica.cc:330] T a454436a865e42a8b2fdeaccdb2b28f9 P e4742b727bc24e89ad40347bfc4a8942: stopping tablet replica
I20260619 14:46:31.267725 2076 raft_consensus.cc:2229] T a454436a865e42a8b2fdeaccdb2b28f9 P e4742b727bc24e89ad40347bfc4a8942 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:46:31.267859 2076 raft_consensus.cc:2258] T a454436a865e42a8b2fdeaccdb2b28f9 P e4742b727bc24e89ad40347bfc4a8942 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:46:31.268011 2075 raft_consensus.cc:2258] T a454436a865e42a8b2fdeaccdb2b28f9 P 3a138c74c83a4052816fc214d30ac87d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:46:31.268663 2076 ts_tablet_manager.cc:1905] T a454436a865e42a8b2fdeaccdb2b28f9 P e4742b727bc24e89ad40347bfc4a8942: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:46:31.269068 2075 ts_tablet_manager.cc:1905] T a454436a865e42a8b2fdeaccdb2b28f9 P 3a138c74c83a4052816fc214d30ac87d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:46:31.269402 2074 ts_tablet_manager.cc:1918] T a454436a865e42a8b2fdeaccdb2b28f9 P 5e5d18da021245748c1655fe7fed35b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 14:46:31.269493 2074 log.cc:1196] T a454436a865e42a8b2fdeaccdb2b28f9 P 5e5d18da021245748c1655fe7fed35b1: Deleting WAL directory at /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880383132139-392-0/minicluster-data/ts-0/wal/wals/a454436a865e42a8b2fdeaccdb2b28f9
I20260619 14:46:31.269764 2074 ts_tablet_manager.cc:1939] T a454436a865e42a8b2fdeaccdb2b28f9 P 5e5d18da021245748c1655fe7fed35b1: Deleting consensus metadata
I20260619 14:46:31.272330 2076 ts_tablet_manager.cc:1918] T a454436a865e42a8b2fdeaccdb2b28f9 P e4742b727bc24e89ad40347bfc4a8942: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 14:46:31.272459 2076 log.cc:1196] T a454436a865e42a8b2fdeaccdb2b28f9 P e4742b727bc24e89ad40347bfc4a8942: Deleting WAL directory at /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880383132139-392-0/minicluster-data/ts-1/wal/wals/a454436a865e42a8b2fdeaccdb2b28f9
I20260619 14:46:31.273065 2076 ts_tablet_manager.cc:1939] T a454436a865e42a8b2fdeaccdb2b28f9 P e4742b727bc24e89ad40347bfc4a8942: Deleting consensus metadata
I20260619 14:46:31.274418 1492 catalog_manager.cc:4841] TS e4742b727bc24e89ad40347bfc4a8942 (127.0.98.2:38555): tablet a454436a865e42a8b2fdeaccdb2b28f9 (table test-table [id=d01f16fa84aa4068a362f45526cd0e19]) successfully deleted
I20260619 14:46:31.274735 2075 ts_tablet_manager.cc:1918] T a454436a865e42a8b2fdeaccdb2b28f9 P 3a138c74c83a4052816fc214d30ac87d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 14:46:31.275177 2075 log.cc:1196] T a454436a865e42a8b2fdeaccdb2b28f9 P 3a138c74c83a4052816fc214d30ac87d: Deleting WAL directory at /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781880383132139-392-0/minicluster-data/ts-2/wal/wals/a454436a865e42a8b2fdeaccdb2b28f9
I20260619 14:46:31.275302 1493 catalog_manager.cc:4841] TS 5e5d18da021245748c1655fe7fed35b1 (127.0.98.1:46285): tablet a454436a865e42a8b2fdeaccdb2b28f9 (table test-table [id=d01f16fa84aa4068a362f45526cd0e19]) successfully deleted
I20260619 14:46:31.275743 2075 ts_tablet_manager.cc:1939] T a454436a865e42a8b2fdeaccdb2b28f9 P 3a138c74c83a4052816fc214d30ac87d: Deleting consensus metadata
I20260619 14:46:31.276995 1493 catalog_manager.cc:4841] TS 3a138c74c83a4052816fc214d30ac87d (127.0.98.3:41623): tablet a454436a865e42a8b2fdeaccdb2b28f9 (table test-table [id=d01f16fa84aa4068a362f45526cd0e19]) successfully deleted
Jun 19 14:46:31 dist-test-slave-2sb1 krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880390, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 14:46:31.289661 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:31.266216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33686 (local address 127.0.98.1:46285)
0619 14:46:31.266344 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:31.266349 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:31.275098 (+ 8749us) server_negotiation.cc:407] Connection header received
0619 14:46:31.276013 (+ 915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:31.276019 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:31.276094 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:31.276194 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:31.277578 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:31.278528 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:31.279784 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:31.280008 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:31.280101 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:31.284193 (+ 4092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:31.284212 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:31.284232 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:31.284272 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:31.286752 (+ 2480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:31.287417 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:31.287424 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:31.287426 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:31.287546 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:31.287977 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:31.287981 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:31.287983 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:31.288556 (+ 573us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:31.288867 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:31.289329 (+ 462us) server_negotiation.cc:299] Negotiation successful
0619 14:46:31.289492 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260619 14:46:31.290457 1704 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33686
I20260619 14:46:31.303480 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:31.293128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58256 (local address 127.0.98.62:46827)
0619 14:46:31.293266 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:31.293270 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:31.293496 (+ 226us) server_negotiation.cc:407] Connection header received
0619 14:46:31.293640 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:31.293645 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:31.293695 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:31.293789 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:31.294845 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:31.295699 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:31.297004 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:31.297384 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:31.297517 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:31.299399 (+ 1882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:31.299403 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:31.299406 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:31.299440 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:31.301425 (+ 1985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:31.302029 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:31.302033 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:31.302035 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:31.302151 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:31.302471 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:31.302475 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:31.302477 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:31.302747 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:31.302891 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:31.303150 (+ 259us) server_negotiation.cc:299] Negotiation successful
0619 14:46:31.303302 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260619 14:46:31.304248 1507 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58256
I20260619 14:46:31.306275 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 1579
I20260619 14:46:31.316514 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 1719
I20260619 14:46:31.323545 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 1859
I20260619 14:46:31.330986 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 1475
2026-06-19T14:46:31Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3123 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:46:31 dist-test-slave-2sb1 krb5kdc[2083](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:46:31 dist-test-slave-2sb1 krb5kdc[2083](info): set up 2 sockets
Jun 19 14:46:31 dist-test-slave-2sb1 krb5kdc[2083](info): commencing operation
krb5kdc: starting...
W20260619 14:46:32.444802 392 mini_kdc.cc:121] Time spent starting KDC: real 1.089s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/krb5kdc/test-user.keytab.
Jun 19 14:46:32 dist-test-slave-2sb1 krb5kdc[2083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880392, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:46:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:46:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 14:46:32.593678 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38131
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38131
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:46:32.713848 2096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:46:32.714149 2096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:46:32.714191 2096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:46:32.718518 2096 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:46:32.718600 2096 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 14:46:32.718617 2096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:46:32.718636 2096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:46:32.718652 2096 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:46:32.718667 2096 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:46:32.725677 2096 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38131
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38131
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2096
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:46:32.726838 2096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:46:32.727888 2096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:46:32.738416 2101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:32.738416 2102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:32.738499 2104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:32.738909 2096 server_base.cc:1034] running on GCE node
I20260619 14:46:32.739656 2096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:46:32.741083 2096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:46:32.742295 2096 hybrid_clock.cc:648] HybridClock initialized: now 1781880392742266 us; error 50 us; skew 500 ppm
Jun 19 14:46:32 dist-test-slave-2sb1 krb5kdc[2083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880392, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:46:32.745795 2096 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 14:46:32.746970 2096 webserver.cc:460] Webserver started at http://127.0.98.62:35807/ using document root <none> and password file <none>
I20260619 14:46:32.747599 2096 fs_manager.cc:359] Metadata directory not provided
I20260619 14:46:32.747712 2096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:46:32.747957 2096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:46:32.750485 2096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/minicluster-data/master-0/data/instance:
uuid: "13d2cf43b72142a48e518f2725623771"
format_stamp: "Formatted at 2026-06-19 14:46:32 on dist-test-slave-2sb1"
server_key: "71ff61cea9154c41d5b55487fef3908d"
server_key_iv: "e10f72526f72b0dde07499c0a85fc6d4"
server_key_version: "encryptionkey@0"
I20260619 14:46:32.751432 2096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/minicluster-data/master-0/wal/instance:
uuid: "13d2cf43b72142a48e518f2725623771"
format_stamp: "Formatted at 2026-06-19 14:46:32 on dist-test-slave-2sb1"
server_key: "71ff61cea9154c41d5b55487fef3908d"
server_key_iv: "e10f72526f72b0dde07499c0a85fc6d4"
server_key_version: "encryptionkey@0"
I20260619 14:46:32.756774 2096 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 14:46:32.759955 2111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:32.761988 2096 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:46:32.762200 2096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/minicluster-data/master-0/wal
uuid: "13d2cf43b72142a48e518f2725623771"
format_stamp: "Formatted at 2026-06-19 14:46:32 on dist-test-slave-2sb1"
server_key: "71ff61cea9154c41d5b55487fef3908d"
server_key_iv: "e10f72526f72b0dde07499c0a85fc6d4"
server_key_version: "encryptionkey@0"
I20260619 14:46:32.762357 2096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:46:32.776628 2096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:46:32.780730 2096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:46:32.780992 2096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:46:32.790823 2096 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38131
I20260619 14:46:32.790841 2173 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38131 every 8 connection(s)
I20260619 14:46:32.792076 2096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:46:32.795316 2174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:46:32.802153 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 2096
I20260619 14:46:32.802232 2174 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13d2cf43b72142a48e518f2725623771: Bootstrap starting.
I20260619 14:46:32.802300 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/minicluster-data/master-0/wal/instance
I20260619 14:46:32.802589 392 external_mini_cluster.cc:1428] Setting key 5bd54be4833f666bff9f7eadd4d9baa7
I20260619 14:46:32.805382 2174 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 13d2cf43b72142a48e518f2725623771: Neither blocks nor log segments found. Creating new log.
I20260619 14:46:32.806263 2174 log.cc:824] T 00000000000000000000000000000000 P 13d2cf43b72142a48e518f2725623771: Log is configured to *not* fsync() on all Append() calls
I20260619 14:46:32.808701 2174 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 13d2cf43b72142a48e518f2725623771: No bootstrap required, opened a new log
Jun 19 14:46:32 dist-test-slave-2sb1 krb5kdc[2083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880392, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:46:32.813592 2174 raft_consensus.cc:348] T 00000000000000000000000000000000 P 13d2cf43b72142a48e518f2725623771 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13d2cf43b72142a48e518f2725623771" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38131 } }
I20260619 14:46:32.813915 2174 raft_consensus.cc:374] T 00000000000000000000000000000000 P 13d2cf43b72142a48e518f2725623771 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:46:32.814006 2174 raft_consensus.cc:729] T 00000000000000000000000000000000 P 13d2cf43b72142a48e518f2725623771 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13d2cf43b72142a48e518f2725623771, State: Initialized, Role: FOLLOWER
I20260619 14:46:32.814559 2174 consensus_queue.cc:260] T 00000000000000000000000000000000 P 13d2cf43b72142a48e518f2725623771 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13d2cf43b72142a48e518f2725623771" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38131 } }
I20260619 14:46:32.814770 2174 raft_consensus.cc:388] T 00000000000000000000000000000000 P 13d2cf43b72142a48e518f2725623771 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:46:32.814888 2174 raft_consensus.cc:482] T 00000000000000000000000000000000 P 13d2cf43b72142a48e518f2725623771 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:46:32.815037 2174 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 13d2cf43b72142a48e518f2725623771 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:46:32.816817 2174 raft_consensus.cc:504] T 00000000000000000000000000000000 P 13d2cf43b72142a48e518f2725623771 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13d2cf43b72142a48e518f2725623771" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38131 } }
I20260619 14:46:32.817380 2174 leader_election.cc:302] T 00000000000000000000000000000000 P 13d2cf43b72142a48e518f2725623771 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 13d2cf43b72142a48e518f2725623771; no voters:
I20260619 14:46:32.817791 2174 leader_election.cc:288] T 00000000000000000000000000000000 P 13d2cf43b72142a48e518f2725623771 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:46:32.817876 2179 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 13d2cf43b72142a48e518f2725623771 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:46:32.818094 2179 raft_consensus.cc:686] T 00000000000000000000000000000000 P 13d2cf43b72142a48e518f2725623771 [term 1 LEADER]: Becoming Leader. State: Replica: 13d2cf43b72142a48e518f2725623771, State: Running, Role: LEADER
I20260619 14:46:32.818419 2179 consensus_queue.cc:237] T 00000000000000000000000000000000 P 13d2cf43b72142a48e518f2725623771 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13d2cf43b72142a48e518f2725623771" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38131 } }
I20260619 14:46:32.819278 2174 sys_catalog.cc:564] T 00000000000000000000000000000000 P 13d2cf43b72142a48e518f2725623771 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:46:32.820436 2181 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13d2cf43b72142a48e518f2725623771 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "13d2cf43b72142a48e518f2725623771" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13d2cf43b72142a48e518f2725623771" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38131 } } }
I20260619 14:46:32.820587 2181 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13d2cf43b72142a48e518f2725623771 [sys.catalog]: This master's current role is: LEADER
I20260619 14:46:32.820904 2177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:32.804255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33910 (local address 127.0.98.62:38131)
0619 14:46:32.804876 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:32.804891 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:32.804930 (+ 39us) server_negotiation.cc:407] Connection header received
0619 14:46:32.805879 (+ 949us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:32.805901 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:32.806284 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:32.806701 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:32.807564 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:32.808622 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:32.809566 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:32.810116 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:32.810300 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:32.813972 (+ 3672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:32.813980 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:32.814001 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:32.814060 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:32.816510 (+ 2450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:32.817238 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:32.817245 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:32.817254 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:32.817388 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:32.817721 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:32.817725 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:32.817727 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:32.818463 (+ 736us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:32.818707 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:32.819454 (+ 747us) server_negotiation.cc:299] Negotiation successful
0619 14:46:32.819826 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":179,"threads_started":1}
I20260619 14:46:32.821372 2180 sys_catalog.cc:455] T 00000000000000000000000000000000 P 13d2cf43b72142a48e518f2725623771 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 13d2cf43b72142a48e518f2725623771. Latest consensus state: current_term: 1 leader_uuid: "13d2cf43b72142a48e518f2725623771" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "13d2cf43b72142a48e518f2725623771" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38131 } } }
I20260619 14:46:32.821477 2180 sys_catalog.cc:458] T 00000000000000000000000000000000 P 13d2cf43b72142a48e518f2725623771 [sys.catalog]: This master's current role is: LEADER
W20260619 14:46:32.824728 2194 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 13d2cf43b72142a48e518f2725623771: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:46:32.824844 2194 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:46:32.825035 2188 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:46:32.825855 2188 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:46:32.831759 2188 catalog_manager.cc:1347] Generated new cluster ID: f3b5c7a03ea14848b7910ad86ee9c4c5
I20260619 14:46:32.831854 2188 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:46:32.839046 2188 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:46:32.840221 2188 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:46:32.854704 2188 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 13d2cf43b72142a48e518f2725623771: Generated new TSK 0
I20260619 14:46:32.855695 2188 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 14:46:32.896368 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38131
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:46:33.017822 2200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:46:33.018180 2200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:46:33.018311 2200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:46:33.023030 2200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:46:33.023136 2200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:46:33.023209 2200 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 14:46:33.023252 2200 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 14:46:33.023329 2200 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 14:46:33.029918 2200 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:38131
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2200
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:46:33.031103 2200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:46:33.032181 2200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 14:46:33.046154 2200 server_base.cc:1034] running on GCE node
W20260619 14:46:33.046267 2206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:33.046276 2208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:33.046370 2205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:33.046952 2200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:46:33.047868 2200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:46:33.049078 2200 hybrid_clock.cc:648] HybridClock initialized: now 1781880393049057 us; error 35 us; skew 500 ppm
Jun 19 14:46:33 dist-test-slave-2sb1 krb5kdc[2083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880393, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:46:33.052356 2200 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 14:46:33.053558 2200 webserver.cc:460] Webserver started at http://127.0.98.1:41935/ using document root <none> and password file <none>
I20260619 14:46:33.054190 2200 fs_manager.cc:359] Metadata directory not provided
I20260619 14:46:33.054288 2200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:46:33.054554 2200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:46:33.057132 2200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/minicluster-data/ts-0/data/instance:
uuid: "66412a87e1fe45ff8f196b0684154ded"
format_stamp: "Formatted at 2026-06-19 14:46:33 on dist-test-slave-2sb1"
server_key: "081dbaad242909bcd5b79b1269b41574"
server_key_iv: "e705a907b1f797aa56aae8967db50813"
server_key_version: "encryptionkey@0"
I20260619 14:46:33.057830 2200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/minicluster-data/ts-0/wal/instance:
uuid: "66412a87e1fe45ff8f196b0684154ded"
format_stamp: "Formatted at 2026-06-19 14:46:33 on dist-test-slave-2sb1"
server_key: "081dbaad242909bcd5b79b1269b41574"
server_key_iv: "e705a907b1f797aa56aae8967db50813"
server_key_version: "encryptionkey@0"
I20260619 14:46:33.063454 2200 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 14:46:33.066402 2215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:33.068101 2200 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 14:46:33.068320 2200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/minicluster-data/ts-0/wal
uuid: "66412a87e1fe45ff8f196b0684154ded"
format_stamp: "Formatted at 2026-06-19 14:46:33 on dist-test-slave-2sb1"
server_key: "081dbaad242909bcd5b79b1269b41574"
server_key_iv: "e705a907b1f797aa56aae8967db50813"
server_key_version: "encryptionkey@0"
I20260619 14:46:33.068480 2200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:46:33.096431 2200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:46:33.100008 2200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:46:33.100337 2200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:46:33.101871 2200 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:46:33.101926 2200 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:33.101967 2200 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:46:33.102053 2200 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 14:46:33 dist-test-slave-2sb1 krb5kdc[2083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880393, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:46:33.114365 2200 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41829
I20260619 14:46:33.115033 2330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41829 every 8 connection(s)
I20260619 14:46:33.115561 2200 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/minicluster-data/ts-0/data/info.pb
I20260619 14:46:33.118927 2177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:33.103621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53327 (local address 127.0.98.62:38131)
0619 14:46:33.103769 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:33.103772 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:33.104901 (+ 1129us) server_negotiation.cc:407] Connection header received
0619 14:46:33.106371 (+ 1470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:33.106375 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:33.106440 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:33.106556 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:33.108022 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:33.108779 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:33.110099 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:33.110341 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:33.110470 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:33.113510 (+ 3040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:33.113515 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:33.113529 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:33.113560 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:33.115780 (+ 2220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:33.116771 (+ 991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:33.116775 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:33.116777 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:33.116876 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:33.117568 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:33.117571 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:33.117572 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:33.117787 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:33.117885 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:33.118648 (+ 763us) server_negotiation.cc:299] Negotiation successful
0619 14:46:33.118797 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 14:46:33.119787 2224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:33.104002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38131 (local address 127.0.98.1:53327)
0619 14:46:33.104702 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:33.104766 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:33.106103 (+ 1337us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:33.106762 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:33.106769 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:33.107303 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:46:33.107848 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:33.107864 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:33.108915 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:33.108919 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:33.109947 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:33.109956 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:33.110517 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:33.110522 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:33.110712 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:33.111408 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:46:33.111426 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:46:33.113296 (+ 1870us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:46:33.116022 (+ 2726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:33.116028 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:33.116041 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:33.116488 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:33.117059 (+ 571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:33.117062 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:33.117064 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:33.117425 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:33.117918 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:46:33.117924 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:46:33.118182 (+ 258us) client_negotiation.cc:769] Sending connection context
0619 14:46:33.118486 (+ 304us) client_negotiation.cc:240] Negotiation successful
0619 14:46:33.118788 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":154,"threads_started":1}
I20260619 14:46:33.121482 2331 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38131
I20260619 14:46:33.121925 2331 heartbeater.cc:464] Registering TS with master...
I20260619 14:46:33.122741 2331 heartbeater.cc:511] Master 127.0.98.62:38131 requested a full tablet report, sending...
I20260619 14:46:33.124917 2127 ts_manager.cc:194] Registered new tserver with Master: 66412a87e1fe45ff8f196b0684154ded (127.0.98.1:41829)
I20260619 14:46:33.125084 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 2200
I20260619 14:46:33.125216 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/minicluster-data/ts-0/wal/instance
I20260619 14:46:33.125526 392 external_mini_cluster.cc:1428] Setting key 223790870e032396ff9db138439e3f5e
I20260619 14:46:33.126541 2127 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:53327
I20260619 14:46:33.135898 2337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:33.124655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38131 (local address 127.0.98.1:40755)
0619 14:46:33.125126 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:33.125143 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:33.125302 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:33.125623 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:33.125626 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:33.125913 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:46:33.126087 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:33.126099 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:33.127236 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:33.127240 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:33.128173 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:33.128183 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:33.129052 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:33.129055 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:33.129144 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:33.129848 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:46:33.129860 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:46:33.130694 (+ 834us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:46:33.134072 (+ 3378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:33.134075 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:33.134078 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:33.134328 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:33.134723 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:33.134726 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:33.134727 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:33.134816 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:33.135390 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:46:33.135392 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:46:33.135485 (+ 93us) client_negotiation.cc:769] Sending connection context
0619 14:46:33.135585 (+ 100us) client_negotiation.cc:240] Negotiation successful
0619 14:46:33.135729 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":127,"threads_started":1}
I20260619 14:46:33.136067 2177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:33.124758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40755 (local address 127.0.98.62:38131)
0619 14:46:33.124911 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:33.124916 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:33.125219 (+ 303us) server_negotiation.cc:407] Connection header received
0619 14:46:33.125406 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:33.125409 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:33.125456 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:33.125602 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:33.126233 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:33.127029 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:33.128310 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:33.128890 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:33.129034 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:33.130954 (+ 1920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:33.130964 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:33.130967 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:33.130998 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:33.133898 (+ 2900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:33.134450 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:33.134457 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:33.134459 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:33.134560 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:33.134942 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:33.134948 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:33.134952 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:33.135256 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:33.135388 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:33.135698 (+ 310us) server_negotiation.cc:299] Negotiation successful
0619 14:46:33.135850 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":59}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 14:46:33.166522 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38131
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:46:33.288686 2340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:46:33.289254 2340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:46:33.289395 2340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:46:33.293838 2340 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:46:33.293954 2340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:46:33.294034 2340 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 14:46:33.294090 2340 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 14:46:33.294178 2340 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 14:46:33.300794 2340 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:38131
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2340
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:46:33.302656 2340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:46:33.303834 2340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:46:33.315214 2346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:33.315187 2345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:33.315351 2348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:33.315519 2340 server_base.cc:1034] running on GCE node
I20260619 14:46:33.316179 2340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:46:33.317098 2340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:46:33.318276 2340 hybrid_clock.cc:648] HybridClock initialized: now 1781880393318239 us; error 56 us; skew 500 ppm
Jun 19 14:46:33 dist-test-slave-2sb1 krb5kdc[2083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880393, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:46:33.337365 2340 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 14:46:33.338510 2340 webserver.cc:460] Webserver started at http://127.0.98.2:40939/ using document root <none> and password file <none>
I20260619 14:46:33.339154 2340 fs_manager.cc:359] Metadata directory not provided
I20260619 14:46:33.339221 2340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:46:33.339455 2340 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:46:33.341774 2340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/minicluster-data/ts-1/data/instance:
uuid: "2339236c64244c94929ddd818b718bfd"
format_stamp: "Formatted at 2026-06-19 14:46:33 on dist-test-slave-2sb1"
server_key: "b62e6f9bb98caa5451a24559c4e457ae"
server_key_iv: "f78f89b6199693e1853bb5b312932636"
server_key_version: "encryptionkey@0"
I20260619 14:46:33.342367 2340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/minicluster-data/ts-1/wal/instance:
uuid: "2339236c64244c94929ddd818b718bfd"
format_stamp: "Formatted at 2026-06-19 14:46:33 on dist-test-slave-2sb1"
server_key: "b62e6f9bb98caa5451a24559c4e457ae"
server_key_iv: "f78f89b6199693e1853bb5b312932636"
server_key_version: "encryptionkey@0"
I20260619 14:46:33.346616 2340 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 14:46:33.350682 2355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:33.352936 2340 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.001s
I20260619 14:46:33.353140 2340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/minicluster-data/ts-1/wal
uuid: "2339236c64244c94929ddd818b718bfd"
format_stamp: "Formatted at 2026-06-19 14:46:33 on dist-test-slave-2sb1"
server_key: "b62e6f9bb98caa5451a24559c4e457ae"
server_key_iv: "f78f89b6199693e1853bb5b312932636"
server_key_version: "encryptionkey@0"
I20260619 14:46:33.353293 2340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:46:33.400120 2340 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:46:33.404192 2340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:46:33.404426 2340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:46:33.406147 2340 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:46:33.406201 2340 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:33.406245 2340 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:46:33.406306 2340 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 14:46:33 dist-test-slave-2sb1 krb5kdc[2083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880393, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:46:33.419080 2340 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39357
I20260619 14:46:33.419178 2470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39357 every 8 connection(s)
I20260619 14:46:33.420157 2340 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/minicluster-data/ts-1/data/info.pb
I20260619 14:46:33.424176 2177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:33.407353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59139 (local address 127.0.98.62:38131)
0619 14:46:33.407488 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:33.407491 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:33.408637 (+ 1146us) server_negotiation.cc:407] Connection header received
0619 14:46:33.409928 (+ 1291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:33.409934 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:33.410008 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:33.410122 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:33.411669 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:33.412710 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:33.414984 (+ 2274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:33.415398 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:33.415538 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:33.419515 (+ 3977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:33.419520 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:33.419523 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:33.419600 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:33.421546 (+ 1946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:33.422331 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:33.422335 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:33.422337 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:33.422457 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:33.422828 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:33.422832 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:33.422834 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:33.423178 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:33.423315 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:33.423848 (+ 533us) server_negotiation.cc:299] Negotiation successful
0619 14:46:33.423982 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 14:46:33.425129 2364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:33.407899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38131 (local address 127.0.98.2:59139)
0619 14:46:33.408458 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:33.408503 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:33.409668 (+ 1165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:33.410349 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:33.410377 (+ 28us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:33.410812 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:46:33.411458 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:33.411480 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:33.412914 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:33.412921 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:33.414789 (+ 1868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:33.414807 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:33.415596 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:33.415604 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:33.415869 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:33.416712 (+ 843us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:46:33.416741 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:46:33.419286 (+ 2545us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:46:33.421763 (+ 2477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:33.421770 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:33.421805 (+ 35us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:33.422207 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:33.422560 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:33.422562 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:33.422563 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:33.422700 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:33.423309 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:46:33.423316 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:46:33.423593 (+ 277us) client_negotiation.cc:769] Sending connection context
0619 14:46:33.423795 (+ 202us) client_negotiation.cc:240] Negotiation successful
0619 14:46:33.424151 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":160,"threads_started":1}
I20260619 14:46:33.425904 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 2340
I20260619 14:46:33.426016 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/minicluster-data/ts-1/wal/instance
I20260619 14:46:33.426348 392 external_mini_cluster.cc:1428] Setting key 9c0445b193a6807e7b886f73eece7d84
I20260619 14:46:33.427011 2471 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38131
I20260619 14:46:33.427204 2471 heartbeater.cc:464] Registering TS with master...
I20260619 14:46:33.427965 2471 heartbeater.cc:511] Master 127.0.98.62:38131 requested a full tablet report, sending...
I20260619 14:46:33.429251 2128 ts_manager.cc:194] Registered new tserver with Master: 2339236c64244c94929ddd818b718bfd (127.0.98.2:39357)
I20260619 14:46:33.430151 2128 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:59139
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 14:46:33.439105 2477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:33.429261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38131 (local address 127.0.98.2:42803)
0619 14:46:33.430083 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:33.430100 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:33.430225 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:33.430498 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:33.430501 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:33.430765 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:46:33.430928 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:33.430937 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:33.431949 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:33.431955 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:33.433056 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:33.433071 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:33.433588 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:33.433595 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:33.433731 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:33.434399 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:46:33.434420 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:46:33.434925 (+ 505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:46:33.437174 (+ 2249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:33.437177 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:33.437180 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:33.437388 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:33.437820 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:33.437823 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:33.437825 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:33.437948 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:33.438529 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:46:33.438532 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:46:33.438684 (+ 152us) client_negotiation.cc:769] Sending connection context
0619 14:46:33.438777 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 14:46:33.438911 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":700,"thread_start_us":159,"threads_started":1}
I20260619 14:46:33.439092 2177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:33.429335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42803 (local address 127.0.98.62:38131)
0619 14:46:33.429447 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:33.429451 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:33.430255 (+ 804us) server_negotiation.cc:407] Connection header received
0619 14:46:33.430309 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:33.430312 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:33.430364 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:33.430463 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:33.431044 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:33.431786 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:33.433228 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:33.433449 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:33.433612 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:33.435132 (+ 1520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:33.435137 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:33.435139 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:33.435165 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:33.437044 (+ 1879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:33.437523 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:33.437527 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:33.437528 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:33.437688 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:33.438161 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:33.438168 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:33.438172 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:33.438427 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:33.438517 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:33.438851 (+ 334us) server_negotiation.cc:299] Negotiation successful
0619 14:46:33.438958 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 14:46:33.460745 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38131
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:46:33.583455 2480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:46:33.583813 2480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:46:33.583905 2480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:46:33.588212 2480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:46:33.588342 2480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:46:33.588426 2480 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260619 14:46:33.588470 2480 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 14:46:33.588541 2480 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 14:46:33.594244 2480 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:38131
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2480
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:46:33.595419 2480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:46:33.596525 2480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:46:33.607684 2485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:33.607788 2480 server_base.cc:1034] running on GCE node
W20260619 14:46:33.607611 2486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:33.607604 2488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:33.608440 2480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:46:33.609200 2480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:46:33.610433 2480 hybrid_clock.cc:648] HybridClock initialized: now 1781880393610382 us; error 63 us; skew 500 ppm
Jun 19 14:46:33 dist-test-slave-2sb1 krb5kdc[2083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880393, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:46:33.614229 2480 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 14:46:33.615470 2480 webserver.cc:460] Webserver started at http://127.0.98.3:39855/ using document root <none> and password file <none>
I20260619 14:46:33.616128 2480 fs_manager.cc:359] Metadata directory not provided
I20260619 14:46:33.616233 2480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:46:33.616478 2480 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:46:33.618932 2480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/minicluster-data/ts-2/data/instance:
uuid: "8bf8f0ae0321449c8d16ca2db42ed2f7"
format_stamp: "Formatted at 2026-06-19 14:46:33 on dist-test-slave-2sb1"
server_key: "b0756038f9c83ef8f818244764821027"
server_key_iv: "c89338fb3389b9bd41d4e55af56de57e"
server_key_version: "encryptionkey@0"
I20260619 14:46:33.619679 2480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/minicluster-data/ts-2/wal/instance:
uuid: "8bf8f0ae0321449c8d16ca2db42ed2f7"
format_stamp: "Formatted at 2026-06-19 14:46:33 on dist-test-slave-2sb1"
server_key: "b0756038f9c83ef8f818244764821027"
server_key_iv: "c89338fb3389b9bd41d4e55af56de57e"
server_key_version: "encryptionkey@0"
I20260619 14:46:33.624509 2480 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260619 14:46:33.628003 2495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:33.629621 2480 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:46:33.629812 2480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/minicluster-data/ts-2/wal
uuid: "8bf8f0ae0321449c8d16ca2db42ed2f7"
format_stamp: "Formatted at 2026-06-19 14:46:33 on dist-test-slave-2sb1"
server_key: "b0756038f9c83ef8f818244764821027"
server_key_iv: "c89338fb3389b9bd41d4e55af56de57e"
server_key_version: "encryptionkey@0"
I20260619 14:46:33.629964 2480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:46:33.656786 2480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:46:33.660758 2480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:46:33.660989 2480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:46:33.662586 2480 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:46:33.662667 2480 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:33.662739 2480 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:46:33.662793 2480 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 14:46:33 dist-test-slave-2sb1 krb5kdc[2083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880393, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:46:33.674042 2480 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34533
I20260619 14:46:33.674130 2610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34533 every 8 connection(s)
I20260619 14:46:33.675104 2480 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/minicluster-data/ts-2/data/info.pb
I20260619 14:46:33.678475 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 2480
I20260619 14:46:33.678623 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/minicluster-data/ts-2/wal/instance
I20260619 14:46:33.679014 392 external_mini_cluster.cc:1428] Setting key 9a5f4a12d3e214d2d2320e6d4ea83a0d
I20260619 14:46:33.679574 2177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:33.663757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48193 (local address 127.0.98.62:38131)
0619 14:46:33.663912 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:33.663917 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:33.664886 (+ 969us) server_negotiation.cc:407] Connection header received
0619 14:46:33.666225 (+ 1339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:33.666230 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:33.666300 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:33.666403 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:33.668151 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:33.669124 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:33.670711 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:33.671089 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:33.671216 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:33.674785 (+ 3569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:33.674789 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:33.674792 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:33.674822 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:33.676885 (+ 2063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:33.677460 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:33.677463 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:33.677464 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:33.677546 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:33.677987 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:33.677991 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:33.677992 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:33.678427 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:33.678553 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:33.679277 (+ 724us) server_negotiation.cc:299] Negotiation successful
0619 14:46:33.679412 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 14:46:33.680775 2504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:33.664211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38131 (local address 127.0.98.3:48193)
0619 14:46:33.664716 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:33.664773 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:33.665944 (+ 1171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:33.666577 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:33.666584 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:33.667188 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:46:33.667968 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:33.667991 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:33.669306 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:33.669311 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:33.670571 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:33.670582 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:33.671366 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:33.671370 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:33.671634 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:33.672366 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:46:33.672390 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:46:33.674539 (+ 2149us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:46:33.677033 (+ 2494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:33.677038 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:33.677052 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:33.677355 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:33.677722 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:33.677725 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:33.677727 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:33.677875 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:33.678694 (+ 819us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:46:33.678703 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:46:33.678975 (+ 272us) client_negotiation.cc:769] Sending connection context
0619 14:46:33.679233 (+ 258us) client_negotiation.cc:240] Negotiation successful
0619 14:46:33.679617 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":134,"threads_started":1}
I20260619 14:46:33.682253 2611 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38131
I20260619 14:46:33.682443 2611 heartbeater.cc:464] Registering TS with master...
I20260619 14:46:33.683043 2611 heartbeater.cc:511] Master 127.0.98.62:38131 requested a full tablet report, sending...
I20260619 14:46:33.684234 2127 ts_manager.cc:194] Registered new tserver with Master: 8bf8f0ae0321449c8d16ca2db42ed2f7 (127.0.98.3:34533)
I20260619 14:46:33.685091 2127 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:48193
I20260619 14:46:33.693616 2177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:33.684927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43687 (local address 127.0.98.62:38131)
0619 14:46:33.685060 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:33.685065 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:33.685412 (+ 347us) server_negotiation.cc:407] Connection header received
0619 14:46:33.685524 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:33.685528 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:33.685563 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:33.685643 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:33.686184 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:33.686942 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:33.687928 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:33.688208 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:33.688300 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:33.689877 (+ 1577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:33.689880 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:33.689882 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:33.689905 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:33.691623 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:33.692189 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:33.692193 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:33.692194 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:33.692312 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:33.692694 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:33.692698 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:33.692703 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:33.692903 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:33.693017 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:33.693320 (+ 303us) server_negotiation.cc:299] Negotiation successful
0619 14:46:33.693444 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 14:46:33.693636 2617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:33.684868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38131 (local address 127.0.98.3:43687)
0619 14:46:33.685292 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:33.685308 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:33.685436 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:33.685683 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:33.685686 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:33.685911 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:46:33.686060 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:33.686067 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:33.687105 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:33.687108 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:33.687806 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:33.687815 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:33.688496 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:33.688499 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:33.688581 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:33.689325 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:46:33.689341 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:46:33.689723 (+ 382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:46:33.691803 (+ 2080us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:33.691806 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:33.691807 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:33.692045 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:33.692478 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:33.692482 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:33.692484 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:33.692573 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:33.693004 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:46:33.693007 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:46:33.693107 (+ 100us) client_negotiation.cc:769] Sending connection context
0619 14:46:33.693262 (+ 155us) client_negotiation.cc:240] Negotiation successful
0619 14:46:33.693432 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"mutex_wait_us":9,"thread_start_us":98,"threads_started":1}
I20260619 14:46:33.693670 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 14:46:33 dist-test-slave-2sb1 krb5kdc[2083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880393, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 14:46:33 dist-test-slave-2sb1 krb5kdc[2083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880393, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:46:33.725134 2177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:33.713578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33918 (local address 127.0.98.62:38131)
0619 14:46:33.713796 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:33.713800 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:33.713812 (+ 12us) server_negotiation.cc:407] Connection header received
0619 14:46:33.713852 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:33.713854 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:33.713892 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:33.713991 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:33.714921 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:33.715447 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:33.716353 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:33.716588 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:33.716763 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:33.720449 (+ 3686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:33.720453 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:33.720456 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:33.720568 (+ 112us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:33.722944 (+ 2376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:33.723602 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:33.723605 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:33.723606 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:33.723697 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:33.723977 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:33.723979 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:33.723980 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:33.724317 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:33.724433 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:33.724902 (+ 469us) server_negotiation.cc:299] Negotiation successful
0619 14:46:33.725009 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
I20260619 14:46:33.729185 2127 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33918:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:46:33.732635 2127 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:46:33.749294 2633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:33.744422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51396 (local address 127.0.98.3:34533)
0619 14:46:33.744989 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:33.745001 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:33.745023 (+ 22us) server_negotiation.cc:407] Connection header received
0619 14:46:33.745089 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:33.745094 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:33.745278 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:33.745447 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:46:33.745640 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:33.746355 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:33.747860 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:33.748549 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:33.748700 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:33.748877 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:33.749035 (+ 158us) server_negotiation.cc:299] Negotiation successful
0619 14:46:33.749134 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":119,"threads_started":1}
I20260619 14:46:33.749284 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:33.744022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34533 (local address 127.0.0.1:51396)
0619 14:46:33.744669 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:33.744683 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:33.744788 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:33.745433 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:33.745436 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:33.745454 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:46:33.745546 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:33.745553 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:33.746476 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:33.746483 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:33.747724 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:33.747734 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:33.748707 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:33.748711 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:33.748821 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:33.748864 (+ 43us) client_negotiation.cc:769] Sending connection context
0619 14:46:33.748985 (+ 121us) client_negotiation.cc:240] Negotiation successful
0619 14:46:33.749126 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":101,"threads_started":1}
I20260619 14:46:33.750031 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:33.743127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39357 (local address 127.0.0.1:47526)
0619 14:46:33.744310 (+ 1183us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:33.744322 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:33.744389 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:33.745354 (+ 965us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:33.745357 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:33.745375 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:46:33.745478 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:33.745484 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:33.747087 (+ 1603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:33.747090 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:33.748402 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:33.748408 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:33.749605 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:33.749609 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:33.749705 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:33.749734 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 14:46:33.749805 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 14:46:33.749889 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"mutex_wait_us":104,"thread_start_us":129,"threads_started":1}
I20260619 14:46:33.750365 2632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:33.744047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35320 (local address 127.0.98.1:41829)
0619 14:46:33.744433 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:33.744441 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:33.744467 (+ 26us) server_negotiation.cc:407] Connection header received
0619 14:46:33.744546 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:33.744555 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:33.744827 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:33.744990 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:46:33.745352 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:33.746898 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:33.748572 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:33.749454 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:33.749569 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:33.749708 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:33.750120 (+ 412us) server_negotiation.cc:299] Negotiation successful
0619 14:46:33.750227 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":74,"threads_started":1}
I20260619 14:46:33.750840 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:33.743433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47526 (local address 127.0.98.2:39357)
0619 14:46:33.744763 (+ 1330us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:33.744775 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:33.744799 (+ 24us) server_negotiation.cc:407] Connection header received
0619 14:46:33.744870 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:33.744878 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:33.745056 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:33.745204 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:46:33.745811 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:33.746952 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:33.748540 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:33.749433 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:33.750252 (+ 819us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:33.750419 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:33.750469 (+ 50us) server_negotiation.cc:299] Negotiation successful
0619 14:46:33.750574 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1230,"thread_start_us":190,"threads_started":1}
I20260619 14:46:33.753002 2629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:33.743377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41829 (local address 127.0.0.1:35320)
0619 14:46:33.743957 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:33.744021 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:33.744161 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:33.745002 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:33.745010 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:33.745036 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:46:33.745211 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:33.745218 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:33.747061 (+ 1843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:33.747067 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:33.748391 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:33.748403 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:33.749761 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:33.749763 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:33.749828 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:33.749885 (+ 57us) client_negotiation.cc:769] Sending connection context
0619 14:46:33.749956 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 14:46:33.750029 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"mutex_wait_us":9,"thread_start_us":104,"threads_started":1}
I20260619 14:46:33.753459 2545 tablet_service.cc:1467] Processing CreateTablet for tablet fe4b0f12ed65497a9e05b9bffbefe2a4 (DEFAULT_TABLE table=test-table [id=304244872b734f00bcd446dc27a5c712]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:46:33.753988 2405 tablet_service.cc:1467] Processing CreateTablet for tablet fe4b0f12ed65497a9e05b9bffbefe2a4 (DEFAULT_TABLE table=test-table [id=304244872b734f00bcd446dc27a5c712]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:46:33.754710 2545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe4b0f12ed65497a9e05b9bffbefe2a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:46:33.754958 2405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe4b0f12ed65497a9e05b9bffbefe2a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:46:33.755935 2265 tablet_service.cc:1467] Processing CreateTablet for tablet fe4b0f12ed65497a9e05b9bffbefe2a4 (DEFAULT_TABLE table=test-table [id=304244872b734f00bcd446dc27a5c712]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:46:33.757026 2265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe4b0f12ed65497a9e05b9bffbefe2a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:46:33.764386 2635 tablet_bootstrap.cc:492] T fe4b0f12ed65497a9e05b9bffbefe2a4 P 8bf8f0ae0321449c8d16ca2db42ed2f7: Bootstrap starting.
I20260619 14:46:33.765218 2634 tablet_bootstrap.cc:492] T fe4b0f12ed65497a9e05b9bffbefe2a4 P 2339236c64244c94929ddd818b718bfd: Bootstrap starting.
I20260619 14:46:33.765297 2636 tablet_bootstrap.cc:492] T fe4b0f12ed65497a9e05b9bffbefe2a4 P 66412a87e1fe45ff8f196b0684154ded: Bootstrap starting.
I20260619 14:46:33.767702 2636 tablet_bootstrap.cc:654] T fe4b0f12ed65497a9e05b9bffbefe2a4 P 66412a87e1fe45ff8f196b0684154ded: Neither blocks nor log segments found. Creating new log.
I20260619 14:46:33.768226 2635 tablet_bootstrap.cc:654] T fe4b0f12ed65497a9e05b9bffbefe2a4 P 8bf8f0ae0321449c8d16ca2db42ed2f7: Neither blocks nor log segments found. Creating new log.
I20260619 14:46:33.768553 2636 log.cc:824] T fe4b0f12ed65497a9e05b9bffbefe2a4 P 66412a87e1fe45ff8f196b0684154ded: Log is configured to *not* fsync() on all Append() calls
I20260619 14:46:33.769008 2635 log.cc:824] T fe4b0f12ed65497a9e05b9bffbefe2a4 P 8bf8f0ae0321449c8d16ca2db42ed2f7: Log is configured to *not* fsync() on all Append() calls
I20260619 14:46:33.770815 2635 tablet_bootstrap.cc:492] T fe4b0f12ed65497a9e05b9bffbefe2a4 P 8bf8f0ae0321449c8d16ca2db42ed2f7: No bootstrap required, opened a new log
I20260619 14:46:33.770907 2636 tablet_bootstrap.cc:492] T fe4b0f12ed65497a9e05b9bffbefe2a4 P 66412a87e1fe45ff8f196b0684154ded: No bootstrap required, opened a new log
I20260619 14:46:33.771064 2635 ts_tablet_manager.cc:1397] T fe4b0f12ed65497a9e05b9bffbefe2a4 P 8bf8f0ae0321449c8d16ca2db42ed2f7: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260619 14:46:33.771111 2636 ts_tablet_manager.cc:1397] T fe4b0f12ed65497a9e05b9bffbefe2a4 P 66412a87e1fe45ff8f196b0684154ded: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 14:46:33.771134 2634 tablet_bootstrap.cc:654] T fe4b0f12ed65497a9e05b9bffbefe2a4 P 2339236c64244c94929ddd818b718bfd: Neither blocks nor log segments found. Creating new log.
I20260619 14:46:33.771960 2634 log.cc:824] T fe4b0f12ed65497a9e05b9bffbefe2a4 P 2339236c64244c94929ddd818b718bfd: Log is configured to *not* fsync() on all Append() calls
I20260619 14:46:33.774694 2634 tablet_bootstrap.cc:492] T fe4b0f12ed65497a9e05b9bffbefe2a4 P 2339236c64244c94929ddd818b718bfd: No bootstrap required, opened a new log
I20260619 14:46:33.774927 2634 ts_tablet_manager.cc:1397] T fe4b0f12ed65497a9e05b9bffbefe2a4 P 2339236c64244c94929ddd818b718bfd: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20260619 14:46:33.775024 2635 raft_consensus.cc:348] T fe4b0f12ed65497a9e05b9bffbefe2a4 P 8bf8f0ae0321449c8d16ca2db42ed2f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2339236c64244c94929ddd818b718bfd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39357 } } peers { permanent_uuid: "66412a87e1fe45ff8f196b0684154ded" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41829 } } peers { permanent_uuid: "8bf8f0ae0321449c8d16ca2db42ed2f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34533 } }
I20260619 14:46:33.775310 2635 raft_consensus.cc:374] T fe4b0f12ed65497a9e05b9bffbefe2a4 P 8bf8f0ae0321449c8d16ca2db42ed2f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:46:33.775367 2635 raft_consensus.cc:729] T fe4b0f12ed65497a9e05b9bffbefe2a4 P 8bf8f0ae0321449c8d16ca2db42ed2f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bf8f0ae0321449c8d16ca2db42ed2f7, State: Initialized, Role: FOLLOWER
I20260619 14:46:33.775835 2635 consensus_queue.cc:260] T fe4b0f12ed65497a9e05b9bffbefe2a4 P 8bf8f0ae0321449c8d16ca2db42ed2f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2339236c64244c94929ddd818b718bfd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39357 } } peers { permanent_uuid: "66412a87e1fe45ff8f196b0684154ded" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41829 } } peers { permanent_uuid: "8bf8f0ae0321449c8d16ca2db42ed2f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34533 } }
I20260619 14:46:33.775950 2636 raft_consensus.cc:348] T fe4b0f12ed65497a9e05b9bffbefe2a4 P 66412a87e1fe45ff8f196b0684154ded [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2339236c64244c94929ddd818b718bfd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39357 } } peers { permanent_uuid: "66412a87e1fe45ff8f196b0684154ded" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41829 } } peers { permanent_uuid: "8bf8f0ae0321449c8d16ca2db42ed2f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34533 } }
I20260619 14:46:33.776237 2636 raft_consensus.cc:374] T fe4b0f12ed65497a9e05b9bffbefe2a4 P 66412a87e1fe45ff8f196b0684154ded [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:46:33.776306 2636 raft_consensus.cc:729] T fe4b0f12ed65497a9e05b9bffbefe2a4 P 66412a87e1fe45ff8f196b0684154ded [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66412a87e1fe45ff8f196b0684154ded, State: Initialized, Role: FOLLOWER
I20260619 14:46:33.776758 2611 heartbeater.cc:503] Master 127.0.98.62:38131 was elected leader, sending a full tablet report...
I20260619 14:46:33.776772 2636 consensus_queue.cc:260] T fe4b0f12ed65497a9e05b9bffbefe2a4 P 66412a87e1fe45ff8f196b0684154ded [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2339236c64244c94929ddd818b718bfd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39357 } } peers { permanent_uuid: "66412a87e1fe45ff8f196b0684154ded" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41829 } } peers { permanent_uuid: "8bf8f0ae0321449c8d16ca2db42ed2f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34533 } }
I20260619 14:46:33.776990 2635 ts_tablet_manager.cc:1428] T fe4b0f12ed65497a9e05b9bffbefe2a4 P 8bf8f0ae0321449c8d16ca2db42ed2f7: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260619 14:46:33.777725 2636 ts_tablet_manager.cc:1428] T fe4b0f12ed65497a9e05b9bffbefe2a4 P 66412a87e1fe45ff8f196b0684154ded: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260619 14:46:33.778079 2331 heartbeater.cc:503] Master 127.0.98.62:38131 was elected leader, sending a full tablet report...
I20260619 14:46:33.780083 2634 raft_consensus.cc:348] T fe4b0f12ed65497a9e05b9bffbefe2a4 P 2339236c64244c94929ddd818b718bfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2339236c64244c94929ddd818b718bfd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39357 } } peers { permanent_uuid: "66412a87e1fe45ff8f196b0684154ded" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41829 } } peers { permanent_uuid: "8bf8f0ae0321449c8d16ca2db42ed2f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34533 } }
I20260619 14:46:33.780452 2634 raft_consensus.cc:374] T fe4b0f12ed65497a9e05b9bffbefe2a4 P 2339236c64244c94929ddd818b718bfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:46:33.780519 2634 raft_consensus.cc:729] T fe4b0f12ed65497a9e05b9bffbefe2a4 P 2339236c64244c94929ddd818b718bfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2339236c64244c94929ddd818b718bfd, State: Initialized, Role: FOLLOWER
I20260619 14:46:33.781039 2634 consensus_queue.cc:260] T fe4b0f12ed65497a9e05b9bffbefe2a4 P 2339236c64244c94929ddd818b718bfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2339236c64244c94929ddd818b718bfd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39357 } } peers { permanent_uuid: "66412a87e1fe45ff8f196b0684154ded" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41829 } } peers { permanent_uuid: "8bf8f0ae0321449c8d16ca2db42ed2f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34533 } }
I20260619 14:46:33.781975 2471 heartbeater.cc:503] Master 127.0.98.62:38131 was elected leader, sending a full tablet report...
I20260619 14:46:33.782414 2634 ts_tablet_manager.cc:1428] T fe4b0f12ed65497a9e05b9bffbefe2a4 P 2339236c64244c94929ddd818b718bfd: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
W20260619 14:46:33.867342 2332 tablet.cc:2369] T fe4b0f12ed65497a9e05b9bffbefe2a4 P 66412a87e1fe45ff8f196b0684154ded: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 14:46:33.921718 2472 tablet.cc:2369] T fe4b0f12ed65497a9e05b9bffbefe2a4 P 2339236c64244c94929ddd818b718bfd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 14:46:33.926120 2612 tablet.cc:2369] T fe4b0f12ed65497a9e05b9bffbefe2a4 P 8bf8f0ae0321449c8d16ca2db42ed2f7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:46:33.991137 2642 raft_consensus.cc:482] T fe4b0f12ed65497a9e05b9bffbefe2a4 P 2339236c64244c94929ddd818b718bfd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:46:33.991482 2642 raft_consensus.cc:504] T fe4b0f12ed65497a9e05b9bffbefe2a4 P 2339236c64244c94929ddd818b718bfd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2339236c64244c94929ddd818b718bfd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39357 } } peers { permanent_uuid: "66412a87e1fe45ff8f196b0684154ded" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41829 } } peers { permanent_uuid: "8bf8f0ae0321449c8d16ca2db42ed2f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34533 } }
I20260619 14:46:33.992668 2642 leader_election.cc:288] T fe4b0f12ed65497a9e05b9bffbefe2a4 P 2339236c64244c94929ddd818b718bfd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 66412a87e1fe45ff8f196b0684154ded (127.0.98.1:41829), 8bf8f0ae0321449c8d16ca2db42ed2f7 (127.0.98.3:34533)
I20260619 14:46:33.997290 2633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:33.993140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60747 (local address 127.0.98.3:34533)
0619 14:46:33.993269 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:33.993284 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:33.993464 (+ 180us) server_negotiation.cc:407] Connection header received
0619 14:46:33.993588 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:33.993594 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:33.993653 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:33.993727 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:46:33.994114 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:33.994664 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:33.995877 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:33.996510 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:33.996611 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:33.996786 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:33.997042 (+ 256us) server_negotiation.cc:299] Negotiation successful
0619 14:46:33.997150 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 14:46:33.997315 2644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:33.993021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34533 (local address 127.0.98.2:60747)
0619 14:46:33.993344 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:33.993359 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:33.993476 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:33.993772 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:33.993776 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:33.993798 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:46:33.993985 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:33.993991 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:33.994796 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:33.994800 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:33.995669 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:33.995678 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:33.996776 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:33.996781 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:33.996888 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:33.996917 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 14:46:33.997014 (+ 97us) client_negotiation.cc:240] Negotiation successful
0619 14:46:33.997116 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"mutex_wait_us":44,"thread_start_us":81,"threads_started":1}
I20260619 14:46:33.997793 2632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:33.993182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47443 (local address 127.0.98.1:41829)
0619 14:46:33.993319 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:33.993324 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:33.993560 (+ 236us) server_negotiation.cc:407] Connection header received
0619 14:46:33.993658 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:33.993660 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:33.993728 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:33.993808 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:46:33.994141 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:33.994690 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:33.996335 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:33.997033 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:33.997131 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:33.997188 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:33.997568 (+ 380us) server_negotiation.cc:299] Negotiation successful
0619 14:46:33.997654 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 14:46:33.998157 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:33.993057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41829 (local address 127.0.98.2:47443)
0619 14:46:33.993439 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:33.993455 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:33.993565 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:33.993846 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:33.993849 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:33.993865 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:46:33.993990 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:33.993995 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:33.994840 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:33.994842 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:33.996002 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:33.996027 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:33.997159 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:33.997163 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:33.997396 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:33.997425 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 14:46:33.997963 (+ 538us) client_negotiation.cc:240] Negotiation successful
0619 14:46:33.998050 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"mutex_wait_us":49,"thread_start_us":80,"threads_started":1}
I20260619 14:46:33.998111 2565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe4b0f12ed65497a9e05b9bffbefe2a4" candidate_uuid: "2339236c64244c94929ddd818b718bfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bf8f0ae0321449c8d16ca2db42ed2f7" is_pre_election: true
I20260619 14:46:33.998471 2565 raft_consensus.cc:2454] T fe4b0f12ed65497a9e05b9bffbefe2a4 P 8bf8f0ae0321449c8d16ca2db42ed2f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2339236c64244c94929ddd818b718bfd in term 0.
I20260619 14:46:33.998832 2285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe4b0f12ed65497a9e05b9bffbefe2a4" candidate_uuid: "2339236c64244c94929ddd818b718bfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66412a87e1fe45ff8f196b0684154ded" is_pre_election: true
I20260619 14:46:33.999097 2356 leader_election.cc:302] T fe4b0f12ed65497a9e05b9bffbefe2a4 P 2339236c64244c94929ddd818b718bfd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2339236c64244c94929ddd818b718bfd, 8bf8f0ae0321449c8d16ca2db42ed2f7; no voters:
I20260619 14:46:33.999171 2285 raft_consensus.cc:2454] T fe4b0f12ed65497a9e05b9bffbefe2a4 P 66412a87e1fe45ff8f196b0684154ded [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2339236c64244c94929ddd818b718bfd in term 0.
I20260619 14:46:33.999419 2642 raft_consensus.cc:2781] T fe4b0f12ed65497a9e05b9bffbefe2a4 P 2339236c64244c94929ddd818b718bfd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:46:33.999504 2642 raft_consensus.cc:482] T fe4b0f12ed65497a9e05b9bffbefe2a4 P 2339236c64244c94929ddd818b718bfd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:46:33.999536 2642 raft_consensus.cc:3037] T fe4b0f12ed65497a9e05b9bffbefe2a4 P 2339236c64244c94929ddd818b718bfd [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:46:34.000844 2642 raft_consensus.cc:504] T fe4b0f12ed65497a9e05b9bffbefe2a4 P 2339236c64244c94929ddd818b718bfd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2339236c64244c94929ddd818b718bfd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39357 } } peers { permanent_uuid: "66412a87e1fe45ff8f196b0684154ded" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41829 } } peers { permanent_uuid: "8bf8f0ae0321449c8d16ca2db42ed2f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34533 } }
I20260619 14:46:34.001230 2642 leader_election.cc:288] T fe4b0f12ed65497a9e05b9bffbefe2a4 P 2339236c64244c94929ddd818b718bfd [CANDIDATE]: Term 1 election: Requested vote from peers 66412a87e1fe45ff8f196b0684154ded (127.0.98.1:41829), 8bf8f0ae0321449c8d16ca2db42ed2f7 (127.0.98.3:34533)
I20260619 14:46:34.001677 2565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe4b0f12ed65497a9e05b9bffbefe2a4" candidate_uuid: "2339236c64244c94929ddd818b718bfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bf8f0ae0321449c8d16ca2db42ed2f7"
I20260619 14:46:34.001677 2285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe4b0f12ed65497a9e05b9bffbefe2a4" candidate_uuid: "2339236c64244c94929ddd818b718bfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66412a87e1fe45ff8f196b0684154ded"
I20260619 14:46:34.001847 2285 raft_consensus.cc:3037] T fe4b0f12ed65497a9e05b9bffbefe2a4 P 66412a87e1fe45ff8f196b0684154ded [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:46:34.001839 2565 raft_consensus.cc:3037] T fe4b0f12ed65497a9e05b9bffbefe2a4 P 8bf8f0ae0321449c8d16ca2db42ed2f7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:46:34.003618 2285 raft_consensus.cc:2454] T fe4b0f12ed65497a9e05b9bffbefe2a4 P 66412a87e1fe45ff8f196b0684154ded [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2339236c64244c94929ddd818b718bfd in term 1.
I20260619 14:46:34.003814 2565 raft_consensus.cc:2454] T fe4b0f12ed65497a9e05b9bffbefe2a4 P 8bf8f0ae0321449c8d16ca2db42ed2f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2339236c64244c94929ddd818b718bfd in term 1.
I20260619 14:46:34.004186 2356 leader_election.cc:302] T fe4b0f12ed65497a9e05b9bffbefe2a4 P 2339236c64244c94929ddd818b718bfd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2339236c64244c94929ddd818b718bfd, 8bf8f0ae0321449c8d16ca2db42ed2f7; no voters:
I20260619 14:46:34.004727 2642 raft_consensus.cc:2781] T fe4b0f12ed65497a9e05b9bffbefe2a4 P 2339236c64244c94929ddd818b718bfd [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:46:34.004931 2642 raft_consensus.cc:686] T fe4b0f12ed65497a9e05b9bffbefe2a4 P 2339236c64244c94929ddd818b718bfd [term 1 LEADER]: Becoming Leader. State: Replica: 2339236c64244c94929ddd818b718bfd, State: Running, Role: LEADER
I20260619 14:46:34.005342 2642 consensus_queue.cc:237] T fe4b0f12ed65497a9e05b9bffbefe2a4 P 2339236c64244c94929ddd818b718bfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2339236c64244c94929ddd818b718bfd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39357 } } peers { permanent_uuid: "66412a87e1fe45ff8f196b0684154ded" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41829 } } peers { permanent_uuid: "8bf8f0ae0321449c8d16ca2db42ed2f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34533 } }
I20260619 14:46:34.008841 2128 catalog_manager.cc:5495] T fe4b0f12ed65497a9e05b9bffbefe2a4 P 2339236c64244c94929ddd818b718bfd reported cstate change: term changed from 0 to 1, leader changed from <none> to 2339236c64244c94929ddd818b718bfd (127.0.98.2). New cstate: current_term: 1 leader_uuid: "2339236c64244c94929ddd818b718bfd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2339236c64244c94929ddd818b718bfd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66412a87e1fe45ff8f196b0684154ded" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8bf8f0ae0321449c8d16ca2db42ed2f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34533 } health_report { overall_health: UNKNOWN } } }
Jun 19 14:46:34 dist-test-slave-2sb1 krb5kdc[2083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880392, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:46:34.070129 2652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:34.059002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38131 (local address 127.0.0.1:33934)
0619 14:46:34.059512 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:34.059527 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:34.059684 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:34.060049 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:34.060052 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:34.060596 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:46:34.060810 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:34.060818 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:34.061649 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:34.061655 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:34.062284 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:34.062292 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:34.062805 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:34.062808 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:34.062913 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:34.063721 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:46:34.063735 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:46:34.065515 (+ 1780us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:46:34.068165 (+ 2650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:34.068185 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:34.068191 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:34.068483 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:34.068978 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:34.068980 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:34.068982 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:34.069078 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:34.069543 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:46:34.069549 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:46:34.069671 (+ 122us) client_negotiation.cc:769] Sending connection context
0619 14:46:34.069820 (+ 149us) client_negotiation.cc:240] Negotiation successful
0619 14:46:34.069969 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":254,"threads_started":1}
I20260619 14:46:34.070158 2177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:34.059108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33934 (local address 127.0.98.62:38131)
0619 14:46:34.059428 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:34.059433 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:34.059610 (+ 177us) server_negotiation.cc:407] Connection header received
0619 14:46:34.059783 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:34.059787 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:34.059830 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:34.060035 (+ 205us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:34.060940 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:34.061519 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:34.062417 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:34.062673 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:34.062799 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:34.065828 (+ 3029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:34.065832 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:34.065834 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:34.065858 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:34.067890 (+ 2032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:34.068625 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:34.068629 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:34.068631 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:34.068834 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:34.069207 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:34.069210 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:34.069211 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:34.069411 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:34.069501 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:34.069934 (+ 433us) server_negotiation.cc:299] Negotiation successful
0619 14:46:34.070045 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176}
I20260619 14:46:34.075961 2128 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33934:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 14:46:34.076889 2128 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:46:34.083583 2545 tablet_service.cc:1467] Processing CreateTablet for tablet 4120ab01b3f14ebabce60bd679341680 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a3190b8c2a6a4236a75957c58de1e9b4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 14:46:34.083588 2405 tablet_service.cc:1467] Processing CreateTablet for tablet 4120ab01b3f14ebabce60bd679341680 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a3190b8c2a6a4236a75957c58de1e9b4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 14:46:34.083588 2265 tablet_service.cc:1467] Processing CreateTablet for tablet 4120ab01b3f14ebabce60bd679341680 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a3190b8c2a6a4236a75957c58de1e9b4]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 14:46:34.084390 2545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4120ab01b3f14ebabce60bd679341680. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:46:34.084551 2405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4120ab01b3f14ebabce60bd679341680. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:46:34.084580 2265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4120ab01b3f14ebabce60bd679341680. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:46:34.087270 2635 tablet_bootstrap.cc:492] T 4120ab01b3f14ebabce60bd679341680 P 8bf8f0ae0321449c8d16ca2db42ed2f7: Bootstrap starting.
I20260619 14:46:34.087869 2634 tablet_bootstrap.cc:492] T 4120ab01b3f14ebabce60bd679341680 P 2339236c64244c94929ddd818b718bfd: Bootstrap starting.
I20260619 14:46:34.088577 2635 tablet_bootstrap.cc:654] T 4120ab01b3f14ebabce60bd679341680 P 8bf8f0ae0321449c8d16ca2db42ed2f7: Neither blocks nor log segments found. Creating new log.
I20260619 14:46:34.088824 2636 tablet_bootstrap.cc:492] T 4120ab01b3f14ebabce60bd679341680 P 66412a87e1fe45ff8f196b0684154ded: Bootstrap starting.
I20260619 14:46:34.089536 2634 tablet_bootstrap.cc:654] T 4120ab01b3f14ebabce60bd679341680 P 2339236c64244c94929ddd818b718bfd: Neither blocks nor log segments found. Creating new log.
I20260619 14:46:34.089840 2636 tablet_bootstrap.cc:654] T 4120ab01b3f14ebabce60bd679341680 P 66412a87e1fe45ff8f196b0684154ded: Neither blocks nor log segments found. Creating new log.
I20260619 14:46:34.090291 2635 tablet_bootstrap.cc:492] T 4120ab01b3f14ebabce60bd679341680 P 8bf8f0ae0321449c8d16ca2db42ed2f7: No bootstrap required, opened a new log
I20260619 14:46:34.090425 2635 ts_tablet_manager.cc:1397] T 4120ab01b3f14ebabce60bd679341680 P 8bf8f0ae0321449c8d16ca2db42ed2f7: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20260619 14:46:34.091017 2634 tablet_bootstrap.cc:492] T 4120ab01b3f14ebabce60bd679341680 P 2339236c64244c94929ddd818b718bfd: No bootstrap required, opened a new log
I20260619 14:46:34.090960 2635 raft_consensus.cc:348] T 4120ab01b3f14ebabce60bd679341680 P 8bf8f0ae0321449c8d16ca2db42ed2f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66412a87e1fe45ff8f196b0684154ded" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41829 } } peers { permanent_uuid: "2339236c64244c94929ddd818b718bfd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39357 } } peers { permanent_uuid: "8bf8f0ae0321449c8d16ca2db42ed2f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34533 } }
I20260619 14:46:34.091086 2635 raft_consensus.cc:374] T 4120ab01b3f14ebabce60bd679341680 P 8bf8f0ae0321449c8d16ca2db42ed2f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:46:34.091113 2634 ts_tablet_manager.cc:1397] T 4120ab01b3f14ebabce60bd679341680 P 2339236c64244c94929ddd818b718bfd: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260619 14:46:34.091122 2635 raft_consensus.cc:729] T 4120ab01b3f14ebabce60bd679341680 P 8bf8f0ae0321449c8d16ca2db42ed2f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bf8f0ae0321449c8d16ca2db42ed2f7, State: Initialized, Role: FOLLOWER
I20260619 14:46:34.091285 2635 consensus_queue.cc:260] T 4120ab01b3f14ebabce60bd679341680 P 8bf8f0ae0321449c8d16ca2db42ed2f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66412a87e1fe45ff8f196b0684154ded" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41829 } } peers { permanent_uuid: "2339236c64244c94929ddd818b718bfd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39357 } } peers { permanent_uuid: "8bf8f0ae0321449c8d16ca2db42ed2f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34533 } }
I20260619 14:46:34.091559 2635 ts_tablet_manager.cc:1428] T 4120ab01b3f14ebabce60bd679341680 P 8bf8f0ae0321449c8d16ca2db42ed2f7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 14:46:34.091603 2634 raft_consensus.cc:348] T 4120ab01b3f14ebabce60bd679341680 P 2339236c64244c94929ddd818b718bfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66412a87e1fe45ff8f196b0684154ded" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41829 } } peers { permanent_uuid: "2339236c64244c94929ddd818b718bfd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39357 } } peers { permanent_uuid: "8bf8f0ae0321449c8d16ca2db42ed2f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34533 } }
I20260619 14:46:34.091722 2634 raft_consensus.cc:374] T 4120ab01b3f14ebabce60bd679341680 P 2339236c64244c94929ddd818b718bfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:46:34.091755 2634 raft_consensus.cc:729] T 4120ab01b3f14ebabce60bd679341680 P 2339236c64244c94929ddd818b718bfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2339236c64244c94929ddd818b718bfd, State: Initialized, Role: FOLLOWER
I20260619 14:46:34.091650 2640 tablet_replica.cc:439] T 4120ab01b3f14ebabce60bd679341680 P 8bf8f0ae0321449c8d16ca2db42ed2f7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66412a87e1fe45ff8f196b0684154ded" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41829 } } peers { permanent_uuid: "2339236c64244c94929ddd818b718bfd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39357 } } peers { permanent_uuid: "8bf8f0ae0321449c8d16ca2db42ed2f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34533 } } }
I20260619 14:46:34.091840 2640 tablet_replica.cc:442] T 4120ab01b3f14ebabce60bd679341680 P 8bf8f0ae0321449c8d16ca2db42ed2f7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:46:34.091921 2634 consensus_queue.cc:260] T 4120ab01b3f14ebabce60bd679341680 P 2339236c64244c94929ddd818b718bfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66412a87e1fe45ff8f196b0684154ded" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41829 } } peers { permanent_uuid: "2339236c64244c94929ddd818b718bfd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39357 } } peers { permanent_uuid: "8bf8f0ae0321449c8d16ca2db42ed2f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34533 } }
I20260619 14:46:34.092190 2642 tablet_replica.cc:439] T 4120ab01b3f14ebabce60bd679341680 P 2339236c64244c94929ddd818b718bfd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66412a87e1fe45ff8f196b0684154ded" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41829 } } peers { permanent_uuid: "2339236c64244c94929ddd818b718bfd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39357 } } peers { permanent_uuid: "8bf8f0ae0321449c8d16ca2db42ed2f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34533 } } }
I20260619 14:46:34.092340 2642 tablet_replica.cc:442] T 4120ab01b3f14ebabce60bd679341680 P 2339236c64244c94929ddd818b718bfd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:46:34.092967 2636 tablet_bootstrap.cc:492] T 4120ab01b3f14ebabce60bd679341680 P 66412a87e1fe45ff8f196b0684154ded: No bootstrap required, opened a new log
I20260619 14:46:34.093077 2636 ts_tablet_manager.cc:1397] T 4120ab01b3f14ebabce60bd679341680 P 66412a87e1fe45ff8f196b0684154ded: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260619 14:46:34.093300 2634 ts_tablet_manager.cc:1428] T 4120ab01b3f14ebabce60bd679341680 P 2339236c64244c94929ddd818b718bfd: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260619 14:46:34.093672 2636 raft_consensus.cc:348] T 4120ab01b3f14ebabce60bd679341680 P 66412a87e1fe45ff8f196b0684154ded [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66412a87e1fe45ff8f196b0684154ded" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41829 } } peers { permanent_uuid: "2339236c64244c94929ddd818b718bfd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39357 } } peers { permanent_uuid: "8bf8f0ae0321449c8d16ca2db42ed2f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34533 } }
I20260619 14:46:34.093818 2636 raft_consensus.cc:374] T 4120ab01b3f14ebabce60bd679341680 P 66412a87e1fe45ff8f196b0684154ded [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:46:34.093868 2636 raft_consensus.cc:729] T 4120ab01b3f14ebabce60bd679341680 P 66412a87e1fe45ff8f196b0684154ded [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66412a87e1fe45ff8f196b0684154ded, State: Initialized, Role: FOLLOWER
I20260619 14:46:34.094005 2636 consensus_queue.cc:260] T 4120ab01b3f14ebabce60bd679341680 P 66412a87e1fe45ff8f196b0684154ded [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66412a87e1fe45ff8f196b0684154ded" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41829 } } peers { permanent_uuid: "2339236c64244c94929ddd818b718bfd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39357 } } peers { permanent_uuid: "8bf8f0ae0321449c8d16ca2db42ed2f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34533 } }
I20260619 14:46:34.094283 2636 ts_tablet_manager.cc:1428] T 4120ab01b3f14ebabce60bd679341680 P 66412a87e1fe45ff8f196b0684154ded: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 14:46:34.094295 2641 tablet_replica.cc:439] T 4120ab01b3f14ebabce60bd679341680 P 66412a87e1fe45ff8f196b0684154ded: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66412a87e1fe45ff8f196b0684154ded" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41829 } } peers { permanent_uuid: "2339236c64244c94929ddd818b718bfd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39357 } } peers { permanent_uuid: "8bf8f0ae0321449c8d16ca2db42ed2f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34533 } } }
I20260619 14:46:34.094460 2641 tablet_replica.cc:442] T 4120ab01b3f14ebabce60bd679341680 P 66412a87e1fe45ff8f196b0684154ded: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:46:34.146298 2642 raft_consensus.cc:482] T 4120ab01b3f14ebabce60bd679341680 P 2339236c64244c94929ddd818b718bfd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:46:34.146440 2642 raft_consensus.cc:504] T 4120ab01b3f14ebabce60bd679341680 P 2339236c64244c94929ddd818b718bfd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66412a87e1fe45ff8f196b0684154ded" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41829 } } peers { permanent_uuid: "2339236c64244c94929ddd818b718bfd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39357 } } peers { permanent_uuid: "8bf8f0ae0321449c8d16ca2db42ed2f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34533 } }
I20260619 14:46:34.146883 2642 leader_election.cc:288] T 4120ab01b3f14ebabce60bd679341680 P 2339236c64244c94929ddd818b718bfd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 66412a87e1fe45ff8f196b0684154ded (127.0.98.1:41829), 8bf8f0ae0321449c8d16ca2db42ed2f7 (127.0.98.3:34533)
I20260619 14:46:34.147369 2565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4120ab01b3f14ebabce60bd679341680" candidate_uuid: "2339236c64244c94929ddd818b718bfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bf8f0ae0321449c8d16ca2db42ed2f7" is_pre_election: true
I20260619 14:46:34.147396 2285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4120ab01b3f14ebabce60bd679341680" candidate_uuid: "2339236c64244c94929ddd818b718bfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66412a87e1fe45ff8f196b0684154ded" is_pre_election: true
I20260619 14:46:34.147524 2565 raft_consensus.cc:2454] T 4120ab01b3f14ebabce60bd679341680 P 8bf8f0ae0321449c8d16ca2db42ed2f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2339236c64244c94929ddd818b718bfd in term 0.
I20260619 14:46:34.147610 2285 raft_consensus.cc:2454] T 4120ab01b3f14ebabce60bd679341680 P 66412a87e1fe45ff8f196b0684154ded [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2339236c64244c94929ddd818b718bfd in term 0.
I20260619 14:46:34.147888 2356 leader_election.cc:302] T 4120ab01b3f14ebabce60bd679341680 P 2339236c64244c94929ddd818b718bfd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2339236c64244c94929ddd818b718bfd, 8bf8f0ae0321449c8d16ca2db42ed2f7; no voters:
I20260619 14:46:34.148131 2642 raft_consensus.cc:2781] T 4120ab01b3f14ebabce60bd679341680 P 2339236c64244c94929ddd818b718bfd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:46:34.148187 2642 raft_consensus.cc:482] T 4120ab01b3f14ebabce60bd679341680 P 2339236c64244c94929ddd818b718bfd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:46:34.148214 2642 raft_consensus.cc:3037] T 4120ab01b3f14ebabce60bd679341680 P 2339236c64244c94929ddd818b718bfd [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:46:34.149355 2642 raft_consensus.cc:504] T 4120ab01b3f14ebabce60bd679341680 P 2339236c64244c94929ddd818b718bfd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66412a87e1fe45ff8f196b0684154ded" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41829 } } peers { permanent_uuid: "2339236c64244c94929ddd818b718bfd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39357 } } peers { permanent_uuid: "8bf8f0ae0321449c8d16ca2db42ed2f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34533 } }
I20260619 14:46:34.149760 2642 leader_election.cc:288] T 4120ab01b3f14ebabce60bd679341680 P 2339236c64244c94929ddd818b718bfd [CANDIDATE]: Term 1 election: Requested vote from peers 66412a87e1fe45ff8f196b0684154ded (127.0.98.1:41829), 8bf8f0ae0321449c8d16ca2db42ed2f7 (127.0.98.3:34533)
I20260619 14:46:34.150128 2285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4120ab01b3f14ebabce60bd679341680" candidate_uuid: "2339236c64244c94929ddd818b718bfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66412a87e1fe45ff8f196b0684154ded"
I20260619 14:46:34.150249 2285 raft_consensus.cc:3037] T 4120ab01b3f14ebabce60bd679341680 P 66412a87e1fe45ff8f196b0684154ded [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:46:34.150542 2565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4120ab01b3f14ebabce60bd679341680" candidate_uuid: "2339236c64244c94929ddd818b718bfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bf8f0ae0321449c8d16ca2db42ed2f7"
I20260619 14:46:34.150905 2565 raft_consensus.cc:3037] T 4120ab01b3f14ebabce60bd679341680 P 8bf8f0ae0321449c8d16ca2db42ed2f7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:46:34.152143 2565 raft_consensus.cc:2454] T 4120ab01b3f14ebabce60bd679341680 P 8bf8f0ae0321449c8d16ca2db42ed2f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2339236c64244c94929ddd818b718bfd in term 1.
I20260619 14:46:34.152182 2285 raft_consensus.cc:2454] T 4120ab01b3f14ebabce60bd679341680 P 66412a87e1fe45ff8f196b0684154ded [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2339236c64244c94929ddd818b718bfd in term 1.
I20260619 14:46:34.152554 2356 leader_election.cc:302] T 4120ab01b3f14ebabce60bd679341680 P 2339236c64244c94929ddd818b718bfd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2339236c64244c94929ddd818b718bfd, 8bf8f0ae0321449c8d16ca2db42ed2f7; no voters:
I20260619 14:46:34.152805 2642 raft_consensus.cc:2781] T 4120ab01b3f14ebabce60bd679341680 P 2339236c64244c94929ddd818b718bfd [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:46:34.152897 2642 raft_consensus.cc:686] T 4120ab01b3f14ebabce60bd679341680 P 2339236c64244c94929ddd818b718bfd [term 1 LEADER]: Becoming Leader. State: Replica: 2339236c64244c94929ddd818b718bfd, State: Running, Role: LEADER
I20260619 14:46:34.153126 2642 consensus_queue.cc:237] T 4120ab01b3f14ebabce60bd679341680 P 2339236c64244c94929ddd818b718bfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66412a87e1fe45ff8f196b0684154ded" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41829 } } peers { permanent_uuid: "2339236c64244c94929ddd818b718bfd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39357 } } peers { permanent_uuid: "8bf8f0ae0321449c8d16ca2db42ed2f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34533 } }
I20260619 14:46:34.153946 2645 tablet_replica.cc:439] T 4120ab01b3f14ebabce60bd679341680 P 2339236c64244c94929ddd818b718bfd: TxnStatusTablet state changed. Reason: New leader 2339236c64244c94929ddd818b718bfd. Latest consensus state: current_term: 1 leader_uuid: "2339236c64244c94929ddd818b718bfd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66412a87e1fe45ff8f196b0684154ded" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41829 } } peers { permanent_uuid: "2339236c64244c94929ddd818b718bfd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39357 } } peers { permanent_uuid: "8bf8f0ae0321449c8d16ca2db42ed2f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34533 } } }
I20260619 14:46:34.154196 2645 tablet_replica.cc:442] T 4120ab01b3f14ebabce60bd679341680 P 2339236c64244c94929ddd818b718bfd: This TxnStatusTablet replica's current role is: LEADER
I20260619 14:46:34.154561 2657 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 14:46:34.154799 2657 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 14:46:34.155290 2126 catalog_manager.cc:5495] T 4120ab01b3f14ebabce60bd679341680 P 2339236c64244c94929ddd818b718bfd reported cstate change: term changed from 0 to 1, leader changed from <none> to 2339236c64244c94929ddd818b718bfd (127.0.98.2). New cstate: current_term: 1 leader_uuid: "2339236c64244c94929ddd818b718bfd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66412a87e1fe45ff8f196b0684154ded" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2339236c64244c94929ddd818b718bfd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8bf8f0ae0321449c8d16ca2db42ed2f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34533 } health_report { overall_health: UNKNOWN } } }
I20260619 14:46:34.170471 2652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:34.165903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39357 (local address 127.0.0.1:47542)
0619 14:46:34.166096 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:34.166113 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:34.166240 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:34.166612 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:34.166615 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:34.166621 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 14:46:34.166775 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:34.166781 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:34.167872 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:34.167876 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:34.168559 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:34.168568 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:34.169165 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:34.169168 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:34.169233 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:34.169293 (+ 60us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 14:46:34.170152 (+ 859us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 14:46:34.170176 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 14:46:34.170253 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 14:46:34.170332 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260619 14:46:34.170492 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:34.166087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47542 (local address 127.0.98.2:39357)
0619 14:46:34.166231 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:34.166236 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:34.166254 (+ 18us) server_negotiation.cc:407] Connection header received
0619 14:46:34.166360 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:34.166364 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:34.166434 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:34.166545 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:46:34.166890 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:34.167728 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:34.168704 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:34.169053 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:34.169168 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:34.169439 (+ 271us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:46:34.170003 (+ 564us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:46:34.170125 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:34.170287 (+ 162us) server_negotiation.cc:299] Negotiation successful
0619 14:46:34.170355 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 14:46:34.176878 2565 raft_consensus.cc:1261] T 4120ab01b3f14ebabce60bd679341680 P 8bf8f0ae0321449c8d16ca2db42ed2f7 [term 1 FOLLOWER]: Refusing update from remote peer 2339236c64244c94929ddd818b718bfd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:46:34.176872 2285 raft_consensus.cc:1261] T 4120ab01b3f14ebabce60bd679341680 P 66412a87e1fe45ff8f196b0684154ded [term 1 FOLLOWER]: Refusing update from remote peer 2339236c64244c94929ddd818b718bfd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:46:34.177695 2642 consensus_queue.cc:1035] T 4120ab01b3f14ebabce60bd679341680 P 2339236c64244c94929ddd818b718bfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8bf8f0ae0321449c8d16ca2db42ed2f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:46:34.178006 2645 consensus_queue.cc:1035] T 4120ab01b3f14ebabce60bd679341680 P 2339236c64244c94929ddd818b718bfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "66412a87e1fe45ff8f196b0684154ded" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:46:34.183312 2641 tablet_replica.cc:439] T 4120ab01b3f14ebabce60bd679341680 P 66412a87e1fe45ff8f196b0684154ded: TxnStatusTablet state changed. Reason: New leader 2339236c64244c94929ddd818b718bfd. Latest consensus state: current_term: 1 leader_uuid: "2339236c64244c94929ddd818b718bfd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66412a87e1fe45ff8f196b0684154ded" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41829 } } peers { permanent_uuid: "2339236c64244c94929ddd818b718bfd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39357 } } peers { permanent_uuid: "8bf8f0ae0321449c8d16ca2db42ed2f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34533 } } }
I20260619 14:46:34.183513 2641 tablet_replica.cc:442] T 4120ab01b3f14ebabce60bd679341680 P 66412a87e1fe45ff8f196b0684154ded: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:46:34.183533 2640 tablet_replica.cc:439] T 4120ab01b3f14ebabce60bd679341680 P 8bf8f0ae0321449c8d16ca2db42ed2f7: TxnStatusTablet state changed. Reason: New leader 2339236c64244c94929ddd818b718bfd. Latest consensus state: current_term: 1 leader_uuid: "2339236c64244c94929ddd818b718bfd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66412a87e1fe45ff8f196b0684154ded" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41829 } } peers { permanent_uuid: "2339236c64244c94929ddd818b718bfd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39357 } } peers { permanent_uuid: "8bf8f0ae0321449c8d16ca2db42ed2f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34533 } } }
I20260619 14:46:34.183648 2640 tablet_replica.cc:442] T 4120ab01b3f14ebabce60bd679341680 P 8bf8f0ae0321449c8d16ca2db42ed2f7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:46:34.188004 2661 mvcc.cc:205] Tried to move back new op lower bound from 7298582094540492800 to 7298582094452740096. Current Snapshot: MvccSnapshot[applied={T|T < 7298582094540492800}]
I20260619 14:46:34.189407 2665 tablet_replica.cc:439] T 4120ab01b3f14ebabce60bd679341680 P 2339236c64244c94929ddd818b718bfd: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2339236c64244c94929ddd818b718bfd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66412a87e1fe45ff8f196b0684154ded" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41829 } } peers { permanent_uuid: "2339236c64244c94929ddd818b718bfd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39357 } } peers { permanent_uuid: "8bf8f0ae0321449c8d16ca2db42ed2f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34533 } } }
I20260619 14:46:34.189574 2665 tablet_replica.cc:442] T 4120ab01b3f14ebabce60bd679341680 P 2339236c64244c94929ddd818b718bfd: This TxnStatusTablet replica's current role is: LEADER
I20260619 14:46:34.189764 2642 tablet_replica.cc:439] T 4120ab01b3f14ebabce60bd679341680 P 2339236c64244c94929ddd818b718bfd: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2339236c64244c94929ddd818b718bfd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66412a87e1fe45ff8f196b0684154ded" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41829 } } peers { permanent_uuid: "2339236c64244c94929ddd818b718bfd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39357 } } peers { permanent_uuid: "8bf8f0ae0321449c8d16ca2db42ed2f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34533 } } }
I20260619 14:46:34.189841 2642 tablet_replica.cc:442] T 4120ab01b3f14ebabce60bd679341680 P 2339236c64244c94929ddd818b718bfd: This TxnStatusTablet replica's current role is: LEADER
I20260619 14:46:34.190091 2640 tablet_replica.cc:439] T 4120ab01b3f14ebabce60bd679341680 P 8bf8f0ae0321449c8d16ca2db42ed2f7: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2339236c64244c94929ddd818b718bfd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66412a87e1fe45ff8f196b0684154ded" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41829 } } peers { permanent_uuid: "2339236c64244c94929ddd818b718bfd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39357 } } peers { permanent_uuid: "8bf8f0ae0321449c8d16ca2db42ed2f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34533 } } }
I20260619 14:46:34.190194 2640 tablet_replica.cc:442] T 4120ab01b3f14ebabce60bd679341680 P 8bf8f0ae0321449c8d16ca2db42ed2f7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:46:34.192225 2641 tablet_replica.cc:439] T 4120ab01b3f14ebabce60bd679341680 P 66412a87e1fe45ff8f196b0684154ded: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2339236c64244c94929ddd818b718bfd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66412a87e1fe45ff8f196b0684154ded" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41829 } } peers { permanent_uuid: "2339236c64244c94929ddd818b718bfd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39357 } } peers { permanent_uuid: "8bf8f0ae0321449c8d16ca2db42ed2f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34533 } } }
I20260619 14:46:34.192376 2641 tablet_replica.cc:442] T 4120ab01b3f14ebabce60bd679341680 P 66412a87e1fe45ff8f196b0684154ded: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:46:34.193135 2662 mvcc.cc:205] Tried to move back new op lower bound from 7298582094540492800 to 7298582094452740096. Current Snapshot: MvccSnapshot[applied={T|T < 7298582094540492800}]
I20260619 14:46:34.203016 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:34.199290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47556 (local address 127.0.98.2:39357)
0619 14:46:34.199450 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:34.199455 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:34.199470 (+ 15us) server_negotiation.cc:407] Connection header received
0619 14:46:34.199522 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:34.199525 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:34.199590 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:34.199674 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:46:34.199968 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:34.200802 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:34.201834 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:34.202178 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:34.202282 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:34.202525 (+ 243us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:46:34.202619 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:46:34.202689 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:34.202809 (+ 120us) server_negotiation.cc:299] Negotiation successful
0619 14:46:34.202864 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 14:46:34.216917 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:34.212289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49635 (local address 127.0.98.2:39357)
0619 14:46:34.212418 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:34.212421 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:34.212791 (+ 370us) server_negotiation.cc:407] Connection header received
0619 14:46:34.212983 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:34.212986 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:34.213029 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:34.213276 (+ 247us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:46:34.213551 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:34.214068 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:34.215515 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:34.215750 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:34.215838 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:34.216341 (+ 503us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:46:34.216418 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:46:34.216494 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:34.216755 (+ 261us) server_negotiation.cc:299] Negotiation successful
0619 14:46:34.216809 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 14:46:34.216925 2672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:34.212219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39357 (local address 127.0.98.2:49635)
0619 14:46:34.212632 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:34.212691 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:34.212805 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:34.213295 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:34.213298 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:34.213306 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 14:46:34.213432 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:34.213437 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:34.214261 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:34.214267 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:34.215348 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:34.215360 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:34.215888 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:34.215892 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:34.215994 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:34.216069 (+ 75us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 14:46:34.216520 (+ 451us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 14:46:34.216527 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 14:46:34.216603 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 14:46:34.216720 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"mutex_wait_us":1,"thread_start_us":112,"threads_started":1}
I20260619 14:46:34.225467 2285 raft_consensus.cc:1261] T fe4b0f12ed65497a9e05b9bffbefe2a4 P 66412a87e1fe45ff8f196b0684154ded [term 1 FOLLOWER]: Refusing update from remote peer 2339236c64244c94929ddd818b718bfd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:46:34.225538 2565 raft_consensus.cc:1261] T fe4b0f12ed65497a9e05b9bffbefe2a4 P 8bf8f0ae0321449c8d16ca2db42ed2f7 [term 1 FOLLOWER]: Refusing update from remote peer 2339236c64244c94929ddd818b718bfd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:46:34.226089 2645 consensus_queue.cc:1035] T fe4b0f12ed65497a9e05b9bffbefe2a4 P 2339236c64244c94929ddd818b718bfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8bf8f0ae0321449c8d16ca2db42ed2f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:46:34.226301 2665 consensus_queue.cc:1035] T fe4b0f12ed65497a9e05b9bffbefe2a4 P 2339236c64244c94929ddd818b718bfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "66412a87e1fe45ff8f196b0684154ded" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:46:34.231777 2660 mvcc.cc:205] Tried to move back new op lower bound from 7298582094743285760 to 7298582093848924160. Current Snapshot: MvccSnapshot[applied={T|T < 7298582094743285760}]
I20260619 14:46:34.282886 2126 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33918:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:46:34.283135 2126 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33918:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:46:34.286293 2126 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 13d2cf43b72142a48e518f2725623771: Sending DeleteTablet for 3 replicas of tablet fe4b0f12ed65497a9e05b9bffbefe2a4
I20260619 14:46:34.287314 2545 tablet_service.cc:1514] Processing DeleteTablet for tablet fe4b0f12ed65497a9e05b9bffbefe2a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:46:34 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51396
I20260619 14:46:34.287312 2405 tablet_service.cc:1514] Processing DeleteTablet for tablet fe4b0f12ed65497a9e05b9bffbefe2a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:46:34 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47526
I20260619 14:46:34.288287 2676 tablet_replica.cc:330] T fe4b0f12ed65497a9e05b9bffbefe2a4 P 8bf8f0ae0321449c8d16ca2db42ed2f7: stopping tablet replica
I20260619 14:46:34.288439 2265 tablet_service.cc:1514] Processing DeleteTablet for tablet fe4b0f12ed65497a9e05b9bffbefe2a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:46:34 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35320
I20260619 14:46:34.288615 2126 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33918:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 14:46:34.288763 2676 raft_consensus.cc:2229] T fe4b0f12ed65497a9e05b9bffbefe2a4 P 8bf8f0ae0321449c8d16ca2db42ed2f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:46:34.288990 2676 raft_consensus.cc:2258] T fe4b0f12ed65497a9e05b9bffbefe2a4 P 8bf8f0ae0321449c8d16ca2db42ed2f7 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260619 14:46:34.289125 2126 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:46:34.289072 2677 tablet_replica.cc:330] T fe4b0f12ed65497a9e05b9bffbefe2a4 P 2339236c64244c94929ddd818b718bfd: stopping tablet replica
I20260619 14:46:34.289528 2677 raft_consensus.cc:2229] T fe4b0f12ed65497a9e05b9bffbefe2a4 P 2339236c64244c94929ddd818b718bfd [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:46:34.290258 2677 raft_consensus.cc:2258] T fe4b0f12ed65497a9e05b9bffbefe2a4 P 2339236c64244c94929ddd818b718bfd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:46:34.290632 2678 tablet_replica.cc:330] T fe4b0f12ed65497a9e05b9bffbefe2a4 P 66412a87e1fe45ff8f196b0684154ded: stopping tablet replica
I20260619 14:46:34.290987 2678 raft_consensus.cc:2229] T fe4b0f12ed65497a9e05b9bffbefe2a4 P 66412a87e1fe45ff8f196b0684154ded [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:46:34.291734 2677 ts_tablet_manager.cc:1905] T fe4b0f12ed65497a9e05b9bffbefe2a4 P 2339236c64244c94929ddd818b718bfd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:46:34.292855 2676 ts_tablet_manager.cc:1905] T fe4b0f12ed65497a9e05b9bffbefe2a4 P 8bf8f0ae0321449c8d16ca2db42ed2f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:46:34.295292 2677 ts_tablet_manager.cc:1918] T fe4b0f12ed65497a9e05b9bffbefe2a4 P 2339236c64244c94929ddd818b718bfd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 14:46:34.295425 2677 log.cc:1196] T fe4b0f12ed65497a9e05b9bffbefe2a4 P 2339236c64244c94929ddd818b718bfd: Deleting WAL directory at /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/minicluster-data/ts-1/wal/wals/fe4b0f12ed65497a9e05b9bffbefe2a4
I20260619 14:46:34.295782 2677 ts_tablet_manager.cc:1939] T fe4b0f12ed65497a9e05b9bffbefe2a4 P 2339236c64244c94929ddd818b718bfd: Deleting consensus metadata
I20260619 14:46:34.296723 2676 ts_tablet_manager.cc:1918] T fe4b0f12ed65497a9e05b9bffbefe2a4 P 8bf8f0ae0321449c8d16ca2db42ed2f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 14:46:34.296841 2676 log.cc:1196] T fe4b0f12ed65497a9e05b9bffbefe2a4 P 8bf8f0ae0321449c8d16ca2db42ed2f7: Deleting WAL directory at /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/minicluster-data/ts-2/wal/wals/fe4b0f12ed65497a9e05b9bffbefe2a4
I20260619 14:46:34.297230 2676 ts_tablet_manager.cc:1939] T fe4b0f12ed65497a9e05b9bffbefe2a4 P 8bf8f0ae0321449c8d16ca2db42ed2f7: Deleting consensus metadata
I20260619 14:46:34.297539 2112 catalog_manager.cc:4841] TS 2339236c64244c94929ddd818b718bfd (127.0.98.2:39357): tablet fe4b0f12ed65497a9e05b9bffbefe2a4 (table test-table [id=304244872b734f00bcd446dc27a5c712]) successfully deleted
I20260619 14:46:34.297855 2405 tablet_service.cc:1467] Processing CreateTablet for tablet a30942711581491db786297584bcad4b (DEFAULT_TABLE table=test-table [id=a141d7245a4f46aca7f34d63842e4e9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:46:34.298130 2405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a30942711581491db786297584bcad4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:46:34.298136 2545 tablet_service.cc:1467] Processing CreateTablet for tablet a30942711581491db786297584bcad4b (DEFAULT_TABLE table=test-table [id=a141d7245a4f46aca7f34d63842e4e9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:46:34.298386 2545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a30942711581491db786297584bcad4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:46:34.300398 2635 tablet_bootstrap.cc:492] T a30942711581491db786297584bcad4b P 8bf8f0ae0321449c8d16ca2db42ed2f7: Bootstrap starting.
I20260619 14:46:34.301113 2112 catalog_manager.cc:4841] TS 8bf8f0ae0321449c8d16ca2db42ed2f7 (127.0.98.3:34533): tablet fe4b0f12ed65497a9e05b9bffbefe2a4 (table test-table [id=304244872b734f00bcd446dc27a5c712]) successfully deleted
I20260619 14:46:34.301251 2634 tablet_bootstrap.cc:492] T a30942711581491db786297584bcad4b P 2339236c64244c94929ddd818b718bfd: Bootstrap starting.
I20260619 14:46:34.301541 2635 tablet_bootstrap.cc:654] T a30942711581491db786297584bcad4b P 8bf8f0ae0321449c8d16ca2db42ed2f7: Neither blocks nor log segments found. Creating new log.
I20260619 14:46:34.302420 2634 tablet_bootstrap.cc:654] T a30942711581491db786297584bcad4b P 2339236c64244c94929ddd818b718bfd: Neither blocks nor log segments found. Creating new log.
I20260619 14:46:34.302923 2265 tablet_service.cc:1467] Processing CreateTablet for tablet a30942711581491db786297584bcad4b (DEFAULT_TABLE table=test-table [id=a141d7245a4f46aca7f34d63842e4e9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:46:34.303239 2265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a30942711581491db786297584bcad4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:46:34.303359 2678 raft_consensus.cc:2258] T fe4b0f12ed65497a9e05b9bffbefe2a4 P 66412a87e1fe45ff8f196b0684154ded [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:46:34.305470 2634 tablet_bootstrap.cc:492] T a30942711581491db786297584bcad4b P 2339236c64244c94929ddd818b718bfd: No bootstrap required, opened a new log
I20260619 14:46:34.305593 2634 ts_tablet_manager.cc:1397] T a30942711581491db786297584bcad4b P 2339236c64244c94929ddd818b718bfd: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20260619 14:46:34.306087 2634 raft_consensus.cc:348] T a30942711581491db786297584bcad4b P 2339236c64244c94929ddd818b718bfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66412a87e1fe45ff8f196b0684154ded" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41829 } } peers { permanent_uuid: "2339236c64244c94929ddd818b718bfd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39357 } } peers { permanent_uuid: "8bf8f0ae0321449c8d16ca2db42ed2f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34533 } }
I20260619 14:46:34.306221 2634 raft_consensus.cc:374] T a30942711581491db786297584bcad4b P 2339236c64244c94929ddd818b718bfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:46:34.306259 2634 raft_consensus.cc:729] T a30942711581491db786297584bcad4b P 2339236c64244c94929ddd818b718bfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2339236c64244c94929ddd818b718bfd, State: Initialized, Role: FOLLOWER
I20260619 14:46:34.306293 2636 tablet_bootstrap.cc:492] T a30942711581491db786297584bcad4b P 66412a87e1fe45ff8f196b0684154ded: Bootstrap starting.
I20260619 14:46:34.306386 2634 consensus_queue.cc:260] T a30942711581491db786297584bcad4b P 2339236c64244c94929ddd818b718bfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66412a87e1fe45ff8f196b0684154ded" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41829 } } peers { permanent_uuid: "2339236c64244c94929ddd818b718bfd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39357 } } peers { permanent_uuid: "8bf8f0ae0321449c8d16ca2db42ed2f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34533 } }
I20260619 14:46:34.306460 2678 ts_tablet_manager.cc:1905] T fe4b0f12ed65497a9e05b9bffbefe2a4 P 66412a87e1fe45ff8f196b0684154ded: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:46:34.306674 2634 ts_tablet_manager.cc:1428] T a30942711581491db786297584bcad4b P 2339236c64244c94929ddd818b718bfd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:46:34.307456 2636 tablet_bootstrap.cc:654] T a30942711581491db786297584bcad4b P 66412a87e1fe45ff8f196b0684154ded: Neither blocks nor log segments found. Creating new log.
I20260619 14:46:34.309373 2636 tablet_bootstrap.cc:492] T a30942711581491db786297584bcad4b P 66412a87e1fe45ff8f196b0684154ded: No bootstrap required, opened a new log
I20260619 14:46:34.309505 2636 ts_tablet_manager.cc:1397] T a30942711581491db786297584bcad4b P 66412a87e1fe45ff8f196b0684154ded: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 14:46:34.309768 2635 tablet_bootstrap.cc:492] T a30942711581491db786297584bcad4b P 8bf8f0ae0321449c8d16ca2db42ed2f7: No bootstrap required, opened a new log
I20260619 14:46:34.309871 2635 ts_tablet_manager.cc:1397] T a30942711581491db786297584bcad4b P 8bf8f0ae0321449c8d16ca2db42ed2f7: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20260619 14:46:34.310017 2636 raft_consensus.cc:348] T a30942711581491db786297584bcad4b P 66412a87e1fe45ff8f196b0684154ded [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66412a87e1fe45ff8f196b0684154ded" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41829 } } peers { permanent_uuid: "2339236c64244c94929ddd818b718bfd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39357 } } peers { permanent_uuid: "8bf8f0ae0321449c8d16ca2db42ed2f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34533 } }
I20260619 14:46:34.310132 2636 raft_consensus.cc:374] T a30942711581491db786297584bcad4b P 66412a87e1fe45ff8f196b0684154ded [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:46:34.310164 2636 raft_consensus.cc:729] T a30942711581491db786297584bcad4b P 66412a87e1fe45ff8f196b0684154ded [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66412a87e1fe45ff8f196b0684154ded, State: Initialized, Role: FOLLOWER
I20260619 14:46:34.310321 2636 consensus_queue.cc:260] T a30942711581491db786297584bcad4b P 66412a87e1fe45ff8f196b0684154ded [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66412a87e1fe45ff8f196b0684154ded" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41829 } } peers { permanent_uuid: "2339236c64244c94929ddd818b718bfd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39357 } } peers { permanent_uuid: "8bf8f0ae0321449c8d16ca2db42ed2f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34533 } }
I20260619 14:46:34.310457 2635 raft_consensus.cc:348] T a30942711581491db786297584bcad4b P 8bf8f0ae0321449c8d16ca2db42ed2f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66412a87e1fe45ff8f196b0684154ded" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41829 } } peers { permanent_uuid: "2339236c64244c94929ddd818b718bfd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39357 } } peers { permanent_uuid: "8bf8f0ae0321449c8d16ca2db42ed2f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34533 } }
I20260619 14:46:34.310591 2635 raft_consensus.cc:374] T a30942711581491db786297584bcad4b P 8bf8f0ae0321449c8d16ca2db42ed2f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:46:34.310634 2636 ts_tablet_manager.cc:1428] T a30942711581491db786297584bcad4b P 66412a87e1fe45ff8f196b0684154ded: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 14:46:34.310648 2635 raft_consensus.cc:729] T a30942711581491db786297584bcad4b P 8bf8f0ae0321449c8d16ca2db42ed2f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bf8f0ae0321449c8d16ca2db42ed2f7, State: Initialized, Role: FOLLOWER
I20260619 14:46:34.310803 2635 consensus_queue.cc:260] T a30942711581491db786297584bcad4b P 8bf8f0ae0321449c8d16ca2db42ed2f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66412a87e1fe45ff8f196b0684154ded" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41829 } } peers { permanent_uuid: "2339236c64244c94929ddd818b718bfd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39357 } } peers { permanent_uuid: "8bf8f0ae0321449c8d16ca2db42ed2f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34533 } }
I20260619 14:46:34.311168 2635 ts_tablet_manager.cc:1428] T a30942711581491db786297584bcad4b P 8bf8f0ae0321449c8d16ca2db42ed2f7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:46:34.313843 2678 ts_tablet_manager.cc:1918] T fe4b0f12ed65497a9e05b9bffbefe2a4 P 66412a87e1fe45ff8f196b0684154ded: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 14:46:34.313970 2678 log.cc:1196] T fe4b0f12ed65497a9e05b9bffbefe2a4 P 66412a87e1fe45ff8f196b0684154ded: Deleting WAL directory at /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/minicluster-data/ts-0/wal/wals/fe4b0f12ed65497a9e05b9bffbefe2a4
I20260619 14:46:34.314415 2678 ts_tablet_manager.cc:1939] T fe4b0f12ed65497a9e05b9bffbefe2a4 P 66412a87e1fe45ff8f196b0684154ded: Deleting consensus metadata
I20260619 14:46:34.316036 2114 catalog_manager.cc:4841] TS 66412a87e1fe45ff8f196b0684154ded (127.0.98.1:41829): tablet fe4b0f12ed65497a9e05b9bffbefe2a4 (table test-table [id=304244872b734f00bcd446dc27a5c712]) successfully deleted
I20260619 14:46:34.547113 2641 raft_consensus.cc:482] T a30942711581491db786297584bcad4b P 66412a87e1fe45ff8f196b0684154ded [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:46:34.547255 2641 raft_consensus.cc:504] T a30942711581491db786297584bcad4b P 66412a87e1fe45ff8f196b0684154ded [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66412a87e1fe45ff8f196b0684154ded" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41829 } } peers { permanent_uuid: "2339236c64244c94929ddd818b718bfd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39357 } } peers { permanent_uuid: "8bf8f0ae0321449c8d16ca2db42ed2f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34533 } }
I20260619 14:46:34.548380 2641 leader_election.cc:288] T a30942711581491db786297584bcad4b P 66412a87e1fe45ff8f196b0684154ded [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2339236c64244c94929ddd818b718bfd (127.0.98.2:39357), 8bf8f0ae0321449c8d16ca2db42ed2f7 (127.0.98.3:34533)
I20260619 14:46:34.553644 2684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:34.549360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57831 (local address 127.0.98.3:34533)
0619 14:46:34.549631 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:34.549638 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:34.549776 (+ 138us) server_negotiation.cc:407] Connection header received
0619 14:46:34.549836 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:34.549840 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:34.549892 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:34.549989 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:46:34.550458 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:34.551228 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:34.552371 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:34.552897 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:34.553023 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:34.553099 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:34.553402 (+ 303us) server_negotiation.cc:299] Negotiation successful
0619 14:46:34.553488 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":88,"threads_started":1}
I20260619 14:46:34.553675 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:34.548944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51837 (local address 127.0.98.2:39357)
0619 14:46:34.549311 (+ 367us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:34.549319 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:34.549339 (+ 20us) server_negotiation.cc:407] Connection header received
0619 14:46:34.549405 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:34.549408 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:34.549461 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:34.549552 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:46:34.550037 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:34.550746 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:34.552413 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:34.552905 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:34.553030 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:34.553100 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:34.553475 (+ 375us) server_negotiation.cc:299] Negotiation successful
0619 14:46:34.553558 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260619 14:46:34.553642 2683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:34.549240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34533 (local address 127.0.98.1:57831)
0619 14:46:34.549644 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:34.549657 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:34.549741 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:34.550037 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:34.550040 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:34.550058 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:46:34.550310 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:34.550323 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:34.551368 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:34.551370 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:34.552229 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:34.552238 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:34.553029 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:34.553032 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:34.553262 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:34.553283 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 14:46:34.553352 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 14:46:34.553456 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"mutex_wait_us":22,"thread_start_us":123,"threads_started":1}
I20260619 14:46:34.553664 2682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:34.548788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39357 (local address 127.0.98.1:51837)
0619 14:46:34.549081 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:34.549099 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:34.549222 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:34.549690 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:34.549694 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:34.549716 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:46:34.549863 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:34.549870 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:34.550888 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:34.550891 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:34.552242 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:34.552252 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:34.553082 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:34.553226 (+ 144us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:34.553307 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:34.553325 (+ 18us) client_negotiation.cc:769] Sending connection context
0619 14:46:34.553432 (+ 107us) client_negotiation.cc:240] Negotiation successful
0619 14:46:34.553499 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"mutex_wait_us":38,"spinlock_wait_cycles":7936,"thread_start_us":100,"threads_started":1}
I20260619 14:46:34.554461 2565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a30942711581491db786297584bcad4b" candidate_uuid: "66412a87e1fe45ff8f196b0684154ded" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bf8f0ae0321449c8d16ca2db42ed2f7" is_pre_election: true
I20260619 14:46:34.554494 2424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a30942711581491db786297584bcad4b" candidate_uuid: "66412a87e1fe45ff8f196b0684154ded" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2339236c64244c94929ddd818b718bfd" is_pre_election: true
I20260619 14:46:34.554636 2565 raft_consensus.cc:2454] T a30942711581491db786297584bcad4b P 8bf8f0ae0321449c8d16ca2db42ed2f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66412a87e1fe45ff8f196b0684154ded in term 0.
I20260619 14:46:34.554764 2424 raft_consensus.cc:2454] T a30942711581491db786297584bcad4b P 2339236c64244c94929ddd818b718bfd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66412a87e1fe45ff8f196b0684154ded in term 0.
I20260619 14:46:34.555002 2216 leader_election.cc:302] T a30942711581491db786297584bcad4b P 66412a87e1fe45ff8f196b0684154ded [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66412a87e1fe45ff8f196b0684154ded, 8bf8f0ae0321449c8d16ca2db42ed2f7; no voters:
I20260619 14:46:34.555253 2641 raft_consensus.cc:2781] T a30942711581491db786297584bcad4b P 66412a87e1fe45ff8f196b0684154ded [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:46:34.555311 2641 raft_consensus.cc:482] T a30942711581491db786297584bcad4b P 66412a87e1fe45ff8f196b0684154ded [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:46:34.555339 2641 raft_consensus.cc:3037] T a30942711581491db786297584bcad4b P 66412a87e1fe45ff8f196b0684154ded [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:46:34.556456 2641 raft_consensus.cc:504] T a30942711581491db786297584bcad4b P 66412a87e1fe45ff8f196b0684154ded [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66412a87e1fe45ff8f196b0684154ded" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41829 } } peers { permanent_uuid: "2339236c64244c94929ddd818b718bfd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39357 } } peers { permanent_uuid: "8bf8f0ae0321449c8d16ca2db42ed2f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34533 } }
I20260619 14:46:34.556880 2641 leader_election.cc:288] T a30942711581491db786297584bcad4b P 66412a87e1fe45ff8f196b0684154ded [CANDIDATE]: Term 1 election: Requested vote from peers 2339236c64244c94929ddd818b718bfd (127.0.98.2:39357), 8bf8f0ae0321449c8d16ca2db42ed2f7 (127.0.98.3:34533)
I20260619 14:46:34.557376 2424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a30942711581491db786297584bcad4b" candidate_uuid: "66412a87e1fe45ff8f196b0684154ded" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2339236c64244c94929ddd818b718bfd"
I20260619 14:46:34.557529 2424 raft_consensus.cc:3037] T a30942711581491db786297584bcad4b P 2339236c64244c94929ddd818b718bfd [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:46:34.557521 2565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a30942711581491db786297584bcad4b" candidate_uuid: "66412a87e1fe45ff8f196b0684154ded" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bf8f0ae0321449c8d16ca2db42ed2f7"
I20260619 14:46:34.557778 2565 raft_consensus.cc:3037] T a30942711581491db786297584bcad4b P 8bf8f0ae0321449c8d16ca2db42ed2f7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:46:34.558677 2424 raft_consensus.cc:2454] T a30942711581491db786297584bcad4b P 2339236c64244c94929ddd818b718bfd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66412a87e1fe45ff8f196b0684154ded in term 1.
I20260619 14:46:34.558858 2565 raft_consensus.cc:2454] T a30942711581491db786297584bcad4b P 8bf8f0ae0321449c8d16ca2db42ed2f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66412a87e1fe45ff8f196b0684154ded in term 1.
I20260619 14:46:34.559024 2216 leader_election.cc:302] T a30942711581491db786297584bcad4b P 66412a87e1fe45ff8f196b0684154ded [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2339236c64244c94929ddd818b718bfd, 66412a87e1fe45ff8f196b0684154ded; no voters:
I20260619 14:46:34.559242 2641 raft_consensus.cc:2781] T a30942711581491db786297584bcad4b P 66412a87e1fe45ff8f196b0684154ded [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:46:34.559494 2641 raft_consensus.cc:686] T a30942711581491db786297584bcad4b P 66412a87e1fe45ff8f196b0684154ded [term 1 LEADER]: Becoming Leader. State: Replica: 66412a87e1fe45ff8f196b0684154ded, State: Running, Role: LEADER
I20260619 14:46:34.559774 2641 consensus_queue.cc:237] T a30942711581491db786297584bcad4b P 66412a87e1fe45ff8f196b0684154ded [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66412a87e1fe45ff8f196b0684154ded" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41829 } } peers { permanent_uuid: "2339236c64244c94929ddd818b718bfd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39357 } } peers { permanent_uuid: "8bf8f0ae0321449c8d16ca2db42ed2f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34533 } }
I20260619 14:46:34.562475 2127 catalog_manager.cc:5495] T a30942711581491db786297584bcad4b P 66412a87e1fe45ff8f196b0684154ded reported cstate change: term changed from 0 to 1, leader changed from <none> to 66412a87e1fe45ff8f196b0684154ded (127.0.98.1). New cstate: current_term: 1 leader_uuid: "66412a87e1fe45ff8f196b0684154ded" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66412a87e1fe45ff8f196b0684154ded" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2339236c64244c94929ddd818b718bfd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8bf8f0ae0321449c8d16ca2db42ed2f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34533 } health_report { overall_health: UNKNOWN } } }
I20260619 14:46:34.617295 2687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:34.612527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35322 (local address 127.0.98.1:41829)
0619 14:46:34.612852 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:34.612866 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:34.612878 (+ 12us) server_negotiation.cc:407] Connection header received
0619 14:46:34.612952 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:34.612954 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:34.613008 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:34.613120 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:46:34.613447 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:34.614105 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:34.615122 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:34.615443 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:34.615667 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:34.615807 (+ 140us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:46:34.616445 (+ 638us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:46:34.616552 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:34.616793 (+ 241us) server_negotiation.cc:299] Negotiation successful
0619 14:46:34.616858 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":163,"threads_started":1}
I20260619 14:46:34.630738 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:34.626529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42997 (local address 127.0.98.2:39357)
0619 14:46:34.626655 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:34.626658 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:34.626896 (+ 238us) server_negotiation.cc:407] Connection header received
0619 14:46:34.627050 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:34.627052 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:34.627097 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:34.627169 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:46:34.627720 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:34.628457 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:34.629490 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:34.629745 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:34.629843 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:34.630204 (+ 361us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:46:34.630294 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:46:34.630365 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:34.630547 (+ 182us) server_negotiation.cc:299] Negotiation successful
0619 14:46:34.630616 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 14:46:34.630753 2690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:34.626467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39357 (local address 127.0.98.1:42997)
0619 14:46:34.626829 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:34.626842 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:34.626962 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:34.627368 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:34.627371 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:34.627380 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 14:46:34.627611 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:34.627619 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:34.628572 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:34.628575 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:34.629355 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:34.629366 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:34.629906 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:34.629909 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:34.629992 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:34.630046 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 14:46:34.630411 (+ 365us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 14:46:34.630418 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 14:46:34.630488 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 14:46:34.630565 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":106,"threads_started":1}
I20260619 14:46:34.638818 2565 raft_consensus.cc:1261] T a30942711581491db786297584bcad4b P 8bf8f0ae0321449c8d16ca2db42ed2f7 [term 1 FOLLOWER]: Refusing update from remote peer 66412a87e1fe45ff8f196b0684154ded: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:46:34.639052 2424 raft_consensus.cc:1261] T a30942711581491db786297584bcad4b P 2339236c64244c94929ddd818b718bfd [term 1 FOLLOWER]: Refusing update from remote peer 66412a87e1fe45ff8f196b0684154ded: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:46:34.639497 2641 consensus_queue.cc:1035] T a30942711581491db786297584bcad4b P 66412a87e1fe45ff8f196b0684154ded [LEADER]: Connected to new peer: Peer: permanent_uuid: "8bf8f0ae0321449c8d16ca2db42ed2f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:46:34.639739 2685 consensus_queue.cc:1035] T a30942711581491db786297584bcad4b P 66412a87e1fe45ff8f196b0684154ded [LEADER]: Connected to new peer: Peer: permanent_uuid: "2339236c64244c94929ddd818b718bfd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:46:34.662410 2687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:34.658746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43299 (local address 127.0.98.1:41829)
0619 14:46:34.658858 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:34.658863 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:34.658877 (+ 14us) server_negotiation.cc:407] Connection header received
0619 14:46:34.658992 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:34.658995 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:34.659052 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:34.659139 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:46:34.659396 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:34.659886 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:34.661034 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:34.661452 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:34.661590 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:34.661928 (+ 338us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:46:34.662029 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:46:34.662116 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:34.662208 (+ 92us) server_negotiation.cc:299] Negotiation successful
0619 14:46:34.662266 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260619 14:46:34.662410 2672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:34.658521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41829 (local address 127.0.98.2:43299)
0619 14:46:34.658800 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:34.658817 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:34.658903 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:34.659164 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:34.659167 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:34.659178 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 14:46:34.659287 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:34.659294 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:34.660011 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:34.660014 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:34.660920 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:34.660931 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:34.661605 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:34.661608 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:34.661759 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:34.661789 (+ 30us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 14:46:34.662121 (+ 332us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 14:46:34.662130 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 14:46:34.662200 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 14:46:34.662294 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165}
W20260619 14:46:34.663584 2662 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 14:46:36.230014 2128 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33918:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:46:36.230437 2128 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33918:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:46:36.233475 2128 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 13d2cf43b72142a48e518f2725623771: Sending DeleteTablet for 3 replicas of tablet a30942711581491db786297584bcad4b
I20260619 14:46:36.234207 2265 tablet_service.cc:1514] Processing DeleteTablet for tablet a30942711581491db786297584bcad4b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:46:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35320
I20260619 14:46:36.234441 2405 tablet_service.cc:1514] Processing DeleteTablet for tablet a30942711581491db786297584bcad4b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:46:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47526
I20260619 14:46:36.234653 2545 tablet_service.cc:1514] Processing DeleteTablet for tablet a30942711581491db786297584bcad4b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:46:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51396
I20260619 14:46:36.234990 2718 tablet_replica.cc:330] T a30942711581491db786297584bcad4b P 8bf8f0ae0321449c8d16ca2db42ed2f7: stopping tablet replica
I20260619 14:46:36.235332 2718 raft_consensus.cc:2229] T a30942711581491db786297584bcad4b P 8bf8f0ae0321449c8d16ca2db42ed2f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:46:36.234652 2716 tablet_replica.cc:330] T a30942711581491db786297584bcad4b P 66412a87e1fe45ff8f196b0684154ded: stopping tablet replica
I20260619 14:46:36.235489 2716 raft_consensus.cc:2229] T a30942711581491db786297584bcad4b P 66412a87e1fe45ff8f196b0684154ded [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:46:36.235518 2718 raft_consensus.cc:2258] T a30942711581491db786297584bcad4b P 8bf8f0ae0321449c8d16ca2db42ed2f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:46:36.235988 2716 raft_consensus.cc:2258] T a30942711581491db786297584bcad4b P 66412a87e1fe45ff8f196b0684154ded [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:46:36.236373 2718 ts_tablet_manager.cc:1905] T a30942711581491db786297584bcad4b P 8bf8f0ae0321449c8d16ca2db42ed2f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:46:36.236693 2716 ts_tablet_manager.cc:1905] T a30942711581491db786297584bcad4b P 66412a87e1fe45ff8f196b0684154ded: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:46:36.239215 2718 ts_tablet_manager.cc:1918] T a30942711581491db786297584bcad4b P 8bf8f0ae0321449c8d16ca2db42ed2f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 14:46:36.239317 2718 log.cc:1196] T a30942711581491db786297584bcad4b P 8bf8f0ae0321449c8d16ca2db42ed2f7: Deleting WAL directory at /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/minicluster-data/ts-2/wal/wals/a30942711581491db786297584bcad4b
I20260619 14:46:36.239431 2716 ts_tablet_manager.cc:1918] T a30942711581491db786297584bcad4b P 66412a87e1fe45ff8f196b0684154ded: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 14:46:36.239509 2716 log.cc:1196] T a30942711581491db786297584bcad4b P 66412a87e1fe45ff8f196b0684154ded: Deleting WAL directory at /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/minicluster-data/ts-0/wal/wals/a30942711581491db786297584bcad4b
I20260619 14:46:36.239614 2718 ts_tablet_manager.cc:1939] T a30942711581491db786297584bcad4b P 8bf8f0ae0321449c8d16ca2db42ed2f7: Deleting consensus metadata
I20260619 14:46:36.239784 2716 ts_tablet_manager.cc:1939] T a30942711581491db786297584bcad4b P 66412a87e1fe45ff8f196b0684154ded: Deleting consensus metadata
I20260619 14:46:36.240603 2112 catalog_manager.cc:4841] TS 8bf8f0ae0321449c8d16ca2db42ed2f7 (127.0.98.3:34533): tablet a30942711581491db786297584bcad4b (table test-table [id=a141d7245a4f46aca7f34d63842e4e9a]) successfully deleted
I20260619 14:46:36.240859 2114 catalog_manager.cc:4841] TS 66412a87e1fe45ff8f196b0684154ded (127.0.98.1:41829): tablet a30942711581491db786297584bcad4b (table test-table [id=a141d7245a4f46aca7f34d63842e4e9a]) successfully deleted
I20260619 14:46:36.240899 2717 tablet_replica.cc:330] T a30942711581491db786297584bcad4b P 2339236c64244c94929ddd818b718bfd: stopping tablet replica
I20260619 14:46:36.241082 2717 raft_consensus.cc:2229] T a30942711581491db786297584bcad4b P 2339236c64244c94929ddd818b718bfd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:46:36.241247 2717 raft_consensus.cc:2258] T a30942711581491db786297584bcad4b P 2339236c64244c94929ddd818b718bfd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:46:36.242048 2717 ts_tablet_manager.cc:1905] T a30942711581491db786297584bcad4b P 2339236c64244c94929ddd818b718bfd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:46:36.244948 2717 ts_tablet_manager.cc:1918] T a30942711581491db786297584bcad4b P 2339236c64244c94929ddd818b718bfd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 14:46:36.245062 2717 log.cc:1196] T a30942711581491db786297584bcad4b P 2339236c64244c94929ddd818b718bfd: Deleting WAL directory at /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/minicluster-data/ts-1/wal/wals/a30942711581491db786297584bcad4b
I20260619 14:46:36.245347 2717 ts_tablet_manager.cc:1939] T a30942711581491db786297584bcad4b P 2339236c64244c94929ddd818b718bfd: Deleting consensus metadata
I20260619 14:46:36.246944 2112 catalog_manager.cc:4841] TS 2339236c64244c94929ddd818b718bfd (127.0.98.2:39357): tablet a30942711581491db786297584bcad4b (table test-table [id=a141d7245a4f46aca7f34d63842e4e9a]) successfully deleted
Jun 19 14:46:36 dist-test-slave-2sb1 krb5kdc[2083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880396, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 14:46:36 dist-test-slave-2sb1 krb5kdc[2083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880396, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:46:36.268931 2726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:36.258147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33944 (local address 127.0.98.62:38131)
0619 14:46:36.258493 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:36.258502 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:36.258705 (+ 203us) server_negotiation.cc:407] Connection header received
0619 14:46:36.258878 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:36.258881 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:36.258936 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:36.259039 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:36.259895 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:36.260443 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:36.261426 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:36.261656 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:36.261773 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:36.264925 (+ 3152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:36.264930 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:36.264932 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:36.264961 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:36.266739 (+ 1778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:36.267382 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:36.267388 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:36.267390 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:36.267489 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:36.267882 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:36.267887 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:36.267889 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:36.268105 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:36.268277 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:36.268542 (+ 265us) server_negotiation.cc:299] Negotiation successful
0619 14:46:36.268728 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":112,"threads_started":1}
I20260619 14:46:36.272006 2128 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33944:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:46:36.272519 2128 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:46:36.278370 2265 tablet_service.cc:1467] Processing CreateTablet for tablet 868df187ca5547d38bf15c9f59d0c16d (DEFAULT_TABLE table=test-table [id=ea367c3ad9b249fcb36af8fb3fbdce16]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:46:36.278442 2405 tablet_service.cc:1467] Processing CreateTablet for tablet 868df187ca5547d38bf15c9f59d0c16d (DEFAULT_TABLE table=test-table [id=ea367c3ad9b249fcb36af8fb3fbdce16]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:46:36.278517 2545 tablet_service.cc:1467] Processing CreateTablet for tablet 868df187ca5547d38bf15c9f59d0c16d (DEFAULT_TABLE table=test-table [id=ea367c3ad9b249fcb36af8fb3fbdce16]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:46:36.278725 2265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 868df187ca5547d38bf15c9f59d0c16d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:46:36.278724 2405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 868df187ca5547d38bf15c9f59d0c16d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:46:36.279042 2545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 868df187ca5547d38bf15c9f59d0c16d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:46:36.282071 2728 tablet_bootstrap.cc:492] T 868df187ca5547d38bf15c9f59d0c16d P 66412a87e1fe45ff8f196b0684154ded: Bootstrap starting.
I20260619 14:46:36.283286 2728 tablet_bootstrap.cc:654] T 868df187ca5547d38bf15c9f59d0c16d P 66412a87e1fe45ff8f196b0684154ded: Neither blocks nor log segments found. Creating new log.
I20260619 14:46:36.283993 2727 tablet_bootstrap.cc:492] T 868df187ca5547d38bf15c9f59d0c16d P 2339236c64244c94929ddd818b718bfd: Bootstrap starting.
I20260619 14:46:36.284372 2729 tablet_bootstrap.cc:492] T 868df187ca5547d38bf15c9f59d0c16d P 8bf8f0ae0321449c8d16ca2db42ed2f7: Bootstrap starting.
I20260619 14:46:36.285033 2727 tablet_bootstrap.cc:654] T 868df187ca5547d38bf15c9f59d0c16d P 2339236c64244c94929ddd818b718bfd: Neither blocks nor log segments found. Creating new log.
I20260619 14:46:36.286118 2728 tablet_bootstrap.cc:492] T 868df187ca5547d38bf15c9f59d0c16d P 66412a87e1fe45ff8f196b0684154ded: No bootstrap required, opened a new log
I20260619 14:46:36.286221 2729 tablet_bootstrap.cc:654] T 868df187ca5547d38bf15c9f59d0c16d P 8bf8f0ae0321449c8d16ca2db42ed2f7: Neither blocks nor log segments found. Creating new log.
I20260619 14:46:36.286237 2728 ts_tablet_manager.cc:1397] T 868df187ca5547d38bf15c9f59d0c16d P 66412a87e1fe45ff8f196b0684154ded: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260619 14:46:36.286759 2728 raft_consensus.cc:348] T 868df187ca5547d38bf15c9f59d0c16d P 66412a87e1fe45ff8f196b0684154ded [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66412a87e1fe45ff8f196b0684154ded" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41829 } } peers { permanent_uuid: "2339236c64244c94929ddd818b718bfd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39357 } } peers { permanent_uuid: "8bf8f0ae0321449c8d16ca2db42ed2f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34533 } }
I20260619 14:46:36.286901 2728 raft_consensus.cc:374] T 868df187ca5547d38bf15c9f59d0c16d P 66412a87e1fe45ff8f196b0684154ded [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:46:36.286942 2728 raft_consensus.cc:729] T 868df187ca5547d38bf15c9f59d0c16d P 66412a87e1fe45ff8f196b0684154ded [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66412a87e1fe45ff8f196b0684154ded, State: Initialized, Role: FOLLOWER
I20260619 14:46:36.287101 2728 consensus_queue.cc:260] T 868df187ca5547d38bf15c9f59d0c16d P 66412a87e1fe45ff8f196b0684154ded [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66412a87e1fe45ff8f196b0684154ded" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41829 } } peers { permanent_uuid: "2339236c64244c94929ddd818b718bfd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39357 } } peers { permanent_uuid: "8bf8f0ae0321449c8d16ca2db42ed2f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34533 } }
I20260619 14:46:36.287634 2729 tablet_bootstrap.cc:492] T 868df187ca5547d38bf15c9f59d0c16d P 8bf8f0ae0321449c8d16ca2db42ed2f7: No bootstrap required, opened a new log
I20260619 14:46:36.287736 2729 ts_tablet_manager.cc:1397] T 868df187ca5547d38bf15c9f59d0c16d P 8bf8f0ae0321449c8d16ca2db42ed2f7: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 14:46:36.288211 2729 raft_consensus.cc:348] T 868df187ca5547d38bf15c9f59d0c16d P 8bf8f0ae0321449c8d16ca2db42ed2f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66412a87e1fe45ff8f196b0684154ded" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41829 } } peers { permanent_uuid: "2339236c64244c94929ddd818b718bfd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39357 } } peers { permanent_uuid: "8bf8f0ae0321449c8d16ca2db42ed2f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34533 } }
I20260619 14:46:36.288329 2729 raft_consensus.cc:374] T 868df187ca5547d38bf15c9f59d0c16d P 8bf8f0ae0321449c8d16ca2db42ed2f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:46:36.288362 2729 raft_consensus.cc:729] T 868df187ca5547d38bf15c9f59d0c16d P 8bf8f0ae0321449c8d16ca2db42ed2f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bf8f0ae0321449c8d16ca2db42ed2f7, State: Initialized, Role: FOLLOWER
I20260619 14:46:36.288532 2729 consensus_queue.cc:260] T 868df187ca5547d38bf15c9f59d0c16d P 8bf8f0ae0321449c8d16ca2db42ed2f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66412a87e1fe45ff8f196b0684154ded" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41829 } } peers { permanent_uuid: "2339236c64244c94929ddd818b718bfd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39357 } } peers { permanent_uuid: "8bf8f0ae0321449c8d16ca2db42ed2f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34533 } }
I20260619 14:46:36.289005 2729 ts_tablet_manager.cc:1428] T 868df187ca5547d38bf15c9f59d0c16d P 8bf8f0ae0321449c8d16ca2db42ed2f7: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260619 14:46:36.289299 2728 ts_tablet_manager.cc:1428] T 868df187ca5547d38bf15c9f59d0c16d P 66412a87e1fe45ff8f196b0684154ded: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260619 14:46:36.290928 2727 tablet_bootstrap.cc:492] T 868df187ca5547d38bf15c9f59d0c16d P 2339236c64244c94929ddd818b718bfd: No bootstrap required, opened a new log
I20260619 14:46:36.291049 2727 ts_tablet_manager.cc:1397] T 868df187ca5547d38bf15c9f59d0c16d P 2339236c64244c94929ddd818b718bfd: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260619 14:46:36.291491 2727 raft_consensus.cc:348] T 868df187ca5547d38bf15c9f59d0c16d P 2339236c64244c94929ddd818b718bfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66412a87e1fe45ff8f196b0684154ded" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41829 } } peers { permanent_uuid: "2339236c64244c94929ddd818b718bfd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39357 } } peers { permanent_uuid: "8bf8f0ae0321449c8d16ca2db42ed2f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34533 } }
I20260619 14:46:36.291659 2727 raft_consensus.cc:374] T 868df187ca5547d38bf15c9f59d0c16d P 2339236c64244c94929ddd818b718bfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:46:36.291709 2727 raft_consensus.cc:729] T 868df187ca5547d38bf15c9f59d0c16d P 2339236c64244c94929ddd818b718bfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2339236c64244c94929ddd818b718bfd, State: Initialized, Role: FOLLOWER
I20260619 14:46:36.291873 2727 consensus_queue.cc:260] T 868df187ca5547d38bf15c9f59d0c16d P 2339236c64244c94929ddd818b718bfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66412a87e1fe45ff8f196b0684154ded" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41829 } } peers { permanent_uuid: "2339236c64244c94929ddd818b718bfd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39357 } } peers { permanent_uuid: "8bf8f0ae0321449c8d16ca2db42ed2f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34533 } }
I20260619 14:46:36.292279 2727 ts_tablet_manager.cc:1428] T 868df187ca5547d38bf15c9f59d0c16d P 2339236c64244c94929ddd818b718bfd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:46:36.407648 2702 raft_consensus.cc:482] T 868df187ca5547d38bf15c9f59d0c16d P 2339236c64244c94929ddd818b718bfd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:46:36.407855 2702 raft_consensus.cc:504] T 868df187ca5547d38bf15c9f59d0c16d P 2339236c64244c94929ddd818b718bfd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66412a87e1fe45ff8f196b0684154ded" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41829 } } peers { permanent_uuid: "2339236c64244c94929ddd818b718bfd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39357 } } peers { permanent_uuid: "8bf8f0ae0321449c8d16ca2db42ed2f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34533 } }
I20260619 14:46:36.408528 2702 leader_election.cc:288] T 868df187ca5547d38bf15c9f59d0c16d P 2339236c64244c94929ddd818b718bfd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 66412a87e1fe45ff8f196b0684154ded (127.0.98.1:41829), 8bf8f0ae0321449c8d16ca2db42ed2f7 (127.0.98.3:34533)
I20260619 14:46:36.409230 2285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "868df187ca5547d38bf15c9f59d0c16d" candidate_uuid: "2339236c64244c94929ddd818b718bfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66412a87e1fe45ff8f196b0684154ded" is_pre_election: true
I20260619 14:46:36.409224 2564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "868df187ca5547d38bf15c9f59d0c16d" candidate_uuid: "2339236c64244c94929ddd818b718bfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bf8f0ae0321449c8d16ca2db42ed2f7" is_pre_election: true
I20260619 14:46:36.409418 2285 raft_consensus.cc:2454] T 868df187ca5547d38bf15c9f59d0c16d P 66412a87e1fe45ff8f196b0684154ded [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2339236c64244c94929ddd818b718bfd in term 0.
I20260619 14:46:36.409421 2564 raft_consensus.cc:2454] T 868df187ca5547d38bf15c9f59d0c16d P 8bf8f0ae0321449c8d16ca2db42ed2f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2339236c64244c94929ddd818b718bfd in term 0.
I20260619 14:46:36.409808 2358 leader_election.cc:302] T 868df187ca5547d38bf15c9f59d0c16d P 2339236c64244c94929ddd818b718bfd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2339236c64244c94929ddd818b718bfd, 66412a87e1fe45ff8f196b0684154ded; no voters:
I20260619 14:46:36.410048 2702 raft_consensus.cc:2781] T 868df187ca5547d38bf15c9f59d0c16d P 2339236c64244c94929ddd818b718bfd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:46:36.410122 2702 raft_consensus.cc:482] T 868df187ca5547d38bf15c9f59d0c16d P 2339236c64244c94929ddd818b718bfd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:46:36.410168 2702 raft_consensus.cc:3037] T 868df187ca5547d38bf15c9f59d0c16d P 2339236c64244c94929ddd818b718bfd [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:46:36.411237 2702 raft_consensus.cc:504] T 868df187ca5547d38bf15c9f59d0c16d P 2339236c64244c94929ddd818b718bfd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66412a87e1fe45ff8f196b0684154ded" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41829 } } peers { permanent_uuid: "2339236c64244c94929ddd818b718bfd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39357 } } peers { permanent_uuid: "8bf8f0ae0321449c8d16ca2db42ed2f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34533 } }
I20260619 14:46:36.411607 2702 leader_election.cc:288] T 868df187ca5547d38bf15c9f59d0c16d P 2339236c64244c94929ddd818b718bfd [CANDIDATE]: Term 1 election: Requested vote from peers 66412a87e1fe45ff8f196b0684154ded (127.0.98.1:41829), 8bf8f0ae0321449c8d16ca2db42ed2f7 (127.0.98.3:34533)
I20260619 14:46:36.412004 2285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "868df187ca5547d38bf15c9f59d0c16d" candidate_uuid: "2339236c64244c94929ddd818b718bfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66412a87e1fe45ff8f196b0684154ded"
I20260619 14:46:36.412132 2285 raft_consensus.cc:3037] T 868df187ca5547d38bf15c9f59d0c16d P 66412a87e1fe45ff8f196b0684154ded [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:46:36.412127 2564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "868df187ca5547d38bf15c9f59d0c16d" candidate_uuid: "2339236c64244c94929ddd818b718bfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bf8f0ae0321449c8d16ca2db42ed2f7"
I20260619 14:46:36.412266 2564 raft_consensus.cc:3037] T 868df187ca5547d38bf15c9f59d0c16d P 8bf8f0ae0321449c8d16ca2db42ed2f7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:46:36.413160 2285 raft_consensus.cc:2454] T 868df187ca5547d38bf15c9f59d0c16d P 66412a87e1fe45ff8f196b0684154ded [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2339236c64244c94929ddd818b718bfd in term 1.
I20260619 14:46:36.413404 2564 raft_consensus.cc:2454] T 868df187ca5547d38bf15c9f59d0c16d P 8bf8f0ae0321449c8d16ca2db42ed2f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2339236c64244c94929ddd818b718bfd in term 1.
I20260619 14:46:36.413553 2358 leader_election.cc:302] T 868df187ca5547d38bf15c9f59d0c16d P 2339236c64244c94929ddd818b718bfd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2339236c64244c94929ddd818b718bfd, 66412a87e1fe45ff8f196b0684154ded; no voters:
I20260619 14:46:36.413740 2702 raft_consensus.cc:2781] T 868df187ca5547d38bf15c9f59d0c16d P 2339236c64244c94929ddd818b718bfd [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:46:36.413853 2702 raft_consensus.cc:686] T 868df187ca5547d38bf15c9f59d0c16d P 2339236c64244c94929ddd818b718bfd [term 1 LEADER]: Becoming Leader. State: Replica: 2339236c64244c94929ddd818b718bfd, State: Running, Role: LEADER
I20260619 14:46:36.414008 2702 consensus_queue.cc:237] T 868df187ca5547d38bf15c9f59d0c16d P 2339236c64244c94929ddd818b718bfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66412a87e1fe45ff8f196b0684154ded" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41829 } } peers { permanent_uuid: "2339236c64244c94929ddd818b718bfd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39357 } } peers { permanent_uuid: "8bf8f0ae0321449c8d16ca2db42ed2f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34533 } }
I20260619 14:46:36.415911 2126 catalog_manager.cc:5495] T 868df187ca5547d38bf15c9f59d0c16d P 2339236c64244c94929ddd818b718bfd reported cstate change: term changed from 0 to 1, leader changed from <none> to 2339236c64244c94929ddd818b718bfd (127.0.98.2). New cstate: current_term: 1 leader_uuid: "2339236c64244c94929ddd818b718bfd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66412a87e1fe45ff8f196b0684154ded" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2339236c64244c94929ddd818b718bfd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8bf8f0ae0321449c8d16ca2db42ed2f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34533 } health_report { overall_health: UNKNOWN } } }
I20260619 14:46:36.462929 2736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:36.458991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47558 (local address 127.0.98.2:39357)
0619 14:46:36.459388 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:36.459395 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:36.459409 (+ 14us) server_negotiation.cc:407] Connection header received
0619 14:46:36.459482 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:36.459484 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:36.459671 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:36.459829 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:46:36.460216 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:36.460838 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:36.461762 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:36.462003 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:36.462121 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:36.462403 (+ 282us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:46:36.462483 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:46:36.462569 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:36.462697 (+ 128us) server_negotiation.cc:299] Negotiation successful
0619 14:46:36.462755 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":217,"threads_started":1}
I20260619 14:46:36.469892 2285 raft_consensus.cc:1261] T 868df187ca5547d38bf15c9f59d0c16d P 66412a87e1fe45ff8f196b0684154ded [term 1 FOLLOWER]: Refusing update from remote peer 2339236c64244c94929ddd818b718bfd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:46:36.469892 2564 raft_consensus.cc:1261] T 868df187ca5547d38bf15c9f59d0c16d P 8bf8f0ae0321449c8d16ca2db42ed2f7 [term 1 FOLLOWER]: Refusing update from remote peer 2339236c64244c94929ddd818b718bfd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:46:36.470633 2697 consensus_queue.cc:1035] T 868df187ca5547d38bf15c9f59d0c16d P 2339236c64244c94929ddd818b718bfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8bf8f0ae0321449c8d16ca2db42ed2f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:46:36.470873 2702 consensus_queue.cc:1035] T 868df187ca5547d38bf15c9f59d0c16d P 2339236c64244c94929ddd818b718bfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "66412a87e1fe45ff8f196b0684154ded" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:46:36.508742 2128 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33944:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:46:36.508932 2128 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33944:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:46:36.511335 2128 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 13d2cf43b72142a48e518f2725623771: Sending DeleteTablet for 3 replicas of tablet 868df187ca5547d38bf15c9f59d0c16d
I20260619 14:46:36.512092 2404 tablet_service.cc:1514] Processing DeleteTablet for tablet 868df187ca5547d38bf15c9f59d0c16d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:46:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47526
I20260619 14:46:36.512105 2545 tablet_service.cc:1514] Processing DeleteTablet for tablet 868df187ca5547d38bf15c9f59d0c16d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:46:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51396
I20260619 14:46:36.512300 2717 tablet_replica.cc:330] T 868df187ca5547d38bf15c9f59d0c16d P 2339236c64244c94929ddd818b718bfd: stopping tablet replica
I20260619 14:46:36.512300 2718 tablet_replica.cc:330] T 868df187ca5547d38bf15c9f59d0c16d P 8bf8f0ae0321449c8d16ca2db42ed2f7: stopping tablet replica
I20260619 14:46:36.512199 2265 tablet_service.cc:1514] Processing DeleteTablet for tablet 868df187ca5547d38bf15c9f59d0c16d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:46:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35320
I20260619 14:46:36.512408 2718 raft_consensus.cc:2229] T 868df187ca5547d38bf15c9f59d0c16d P 8bf8f0ae0321449c8d16ca2db42ed2f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:46:36.512408 2717 raft_consensus.cc:2229] T 868df187ca5547d38bf15c9f59d0c16d P 2339236c64244c94929ddd818b718bfd [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:46:36.512526 2718 raft_consensus.cc:2258] T 868df187ca5547d38bf15c9f59d0c16d P 8bf8f0ae0321449c8d16ca2db42ed2f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:46:36.512614 2717 raft_consensus.cc:2258] T 868df187ca5547d38bf15c9f59d0c16d P 2339236c64244c94929ddd818b718bfd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:46:36.512650 2716 tablet_replica.cc:330] T 868df187ca5547d38bf15c9f59d0c16d P 66412a87e1fe45ff8f196b0684154ded: stopping tablet replica
I20260619 14:46:36.512821 2716 raft_consensus.cc:2229] T 868df187ca5547d38bf15c9f59d0c16d P 66412a87e1fe45ff8f196b0684154ded [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:46:36.512965 2716 raft_consensus.cc:2258] T 868df187ca5547d38bf15c9f59d0c16d P 66412a87e1fe45ff8f196b0684154ded [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:46:36.513047 2128 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33944:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:46:36.513511 2128 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:46:36.513885 2717 ts_tablet_manager.cc:1905] T 868df187ca5547d38bf15c9f59d0c16d P 2339236c64244c94929ddd818b718bfd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:46:36.513885 2716 ts_tablet_manager.cc:1905] T 868df187ca5547d38bf15c9f59d0c16d P 66412a87e1fe45ff8f196b0684154ded: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:46:36.513887 2718 ts_tablet_manager.cc:1905] T 868df187ca5547d38bf15c9f59d0c16d P 8bf8f0ae0321449c8d16ca2db42ed2f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:46:36.517118 2718 ts_tablet_manager.cc:1918] T 868df187ca5547d38bf15c9f59d0c16d P 8bf8f0ae0321449c8d16ca2db42ed2f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 14:46:36.517220 2718 log.cc:1196] T 868df187ca5547d38bf15c9f59d0c16d P 8bf8f0ae0321449c8d16ca2db42ed2f7: Deleting WAL directory at /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/minicluster-data/ts-2/wal/wals/868df187ca5547d38bf15c9f59d0c16d
I20260619 14:46:36.517475 2716 ts_tablet_manager.cc:1918] T 868df187ca5547d38bf15c9f59d0c16d P 66412a87e1fe45ff8f196b0684154ded: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 14:46:36.517549 2718 ts_tablet_manager.cc:1939] T 868df187ca5547d38bf15c9f59d0c16d P 8bf8f0ae0321449c8d16ca2db42ed2f7: Deleting consensus metadata
I20260619 14:46:36.517558 2716 log.cc:1196] T 868df187ca5547d38bf15c9f59d0c16d P 66412a87e1fe45ff8f196b0684154ded: Deleting WAL directory at /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/minicluster-data/ts-0/wal/wals/868df187ca5547d38bf15c9f59d0c16d
I20260619 14:46:36.517834 2716 ts_tablet_manager.cc:1939] T 868df187ca5547d38bf15c9f59d0c16d P 66412a87e1fe45ff8f196b0684154ded: Deleting consensus metadata
I20260619 14:46:36.518529 2112 catalog_manager.cc:4841] TS 8bf8f0ae0321449c8d16ca2db42ed2f7 (127.0.98.3:34533): tablet 868df187ca5547d38bf15c9f59d0c16d (table test-table [id=ea367c3ad9b249fcb36af8fb3fbdce16]) successfully deleted
I20260619 14:46:36.518985 2114 catalog_manager.cc:4841] TS 66412a87e1fe45ff8f196b0684154ded (127.0.98.1:41829): tablet 868df187ca5547d38bf15c9f59d0c16d (table test-table [id=ea367c3ad9b249fcb36af8fb3fbdce16]) successfully deleted
I20260619 14:46:36.519765 2717 ts_tablet_manager.cc:1918] T 868df187ca5547d38bf15c9f59d0c16d P 2339236c64244c94929ddd818b718bfd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 14:46:36.519860 2717 log.cc:1196] T 868df187ca5547d38bf15c9f59d0c16d P 2339236c64244c94929ddd818b718bfd: Deleting WAL directory at /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/minicluster-data/ts-1/wal/wals/868df187ca5547d38bf15c9f59d0c16d
I20260619 14:46:36.519837 2265 tablet_service.cc:1467] Processing CreateTablet for tablet 585b46c4b8d24ad89dd4d068eba5c9f8 (DEFAULT_TABLE table=test-table [id=3b92b399f3cc4252b6bc738b3638212d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:46:36.519894 2545 tablet_service.cc:1467] Processing CreateTablet for tablet 585b46c4b8d24ad89dd4d068eba5c9f8 (DEFAULT_TABLE table=test-table [id=3b92b399f3cc4252b6bc738b3638212d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:46:36.520167 2265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 585b46c4b8d24ad89dd4d068eba5c9f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:46:36.520193 2545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 585b46c4b8d24ad89dd4d068eba5c9f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:46:36.520519 2404 tablet_service.cc:1467] Processing CreateTablet for tablet 585b46c4b8d24ad89dd4d068eba5c9f8 (DEFAULT_TABLE table=test-table [id=3b92b399f3cc4252b6bc738b3638212d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:46:36.520769 2717 ts_tablet_manager.cc:1939] T 868df187ca5547d38bf15c9f59d0c16d P 2339236c64244c94929ddd818b718bfd: Deleting consensus metadata
I20260619 14:46:36.520812 2404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 585b46c4b8d24ad89dd4d068eba5c9f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:46:36.522500 2112 catalog_manager.cc:4841] TS 2339236c64244c94929ddd818b718bfd (127.0.98.2:39357): tablet 868df187ca5547d38bf15c9f59d0c16d (table test-table [id=ea367c3ad9b249fcb36af8fb3fbdce16]) successfully deleted
I20260619 14:46:36.523062 2729 tablet_bootstrap.cc:492] T 585b46c4b8d24ad89dd4d068eba5c9f8 P 8bf8f0ae0321449c8d16ca2db42ed2f7: Bootstrap starting.
I20260619 14:46:36.523319 2728 tablet_bootstrap.cc:492] T 585b46c4b8d24ad89dd4d068eba5c9f8 P 66412a87e1fe45ff8f196b0684154ded: Bootstrap starting.
I20260619 14:46:36.524348 2728 tablet_bootstrap.cc:654] T 585b46c4b8d24ad89dd4d068eba5c9f8 P 66412a87e1fe45ff8f196b0684154ded: Neither blocks nor log segments found. Creating new log.
I20260619 14:46:36.524629 2729 tablet_bootstrap.cc:654] T 585b46c4b8d24ad89dd4d068eba5c9f8 P 8bf8f0ae0321449c8d16ca2db42ed2f7: Neither blocks nor log segments found. Creating new log.
I20260619 14:46:36.525563 2727 tablet_bootstrap.cc:492] T 585b46c4b8d24ad89dd4d068eba5c9f8 P 2339236c64244c94929ddd818b718bfd: Bootstrap starting.
I20260619 14:46:36.525949 2728 tablet_bootstrap.cc:492] T 585b46c4b8d24ad89dd4d068eba5c9f8 P 66412a87e1fe45ff8f196b0684154ded: No bootstrap required, opened a new log
I20260619 14:46:36.526027 2728 ts_tablet_manager.cc:1397] T 585b46c4b8d24ad89dd4d068eba5c9f8 P 66412a87e1fe45ff8f196b0684154ded: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260619 14:46:36.526480 2728 raft_consensus.cc:348] T 585b46c4b8d24ad89dd4d068eba5c9f8 P 66412a87e1fe45ff8f196b0684154ded [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66412a87e1fe45ff8f196b0684154ded" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41829 } } peers { permanent_uuid: "2339236c64244c94929ddd818b718bfd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39357 } } peers { permanent_uuid: "8bf8f0ae0321449c8d16ca2db42ed2f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34533 } }
I20260619 14:46:36.526616 2727 tablet_bootstrap.cc:654] T 585b46c4b8d24ad89dd4d068eba5c9f8 P 2339236c64244c94929ddd818b718bfd: Neither blocks nor log segments found. Creating new log.
I20260619 14:46:36.526640 2728 raft_consensus.cc:374] T 585b46c4b8d24ad89dd4d068eba5c9f8 P 66412a87e1fe45ff8f196b0684154ded [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:46:36.526695 2728 raft_consensus.cc:729] T 585b46c4b8d24ad89dd4d068eba5c9f8 P 66412a87e1fe45ff8f196b0684154ded [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66412a87e1fe45ff8f196b0684154ded, State: Initialized, Role: FOLLOWER
I20260619 14:46:36.526851 2728 consensus_queue.cc:260] T 585b46c4b8d24ad89dd4d068eba5c9f8 P 66412a87e1fe45ff8f196b0684154ded [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66412a87e1fe45ff8f196b0684154ded" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41829 } } peers { permanent_uuid: "2339236c64244c94929ddd818b718bfd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39357 } } peers { permanent_uuid: "8bf8f0ae0321449c8d16ca2db42ed2f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34533 } }
I20260619 14:46:36.527156 2728 ts_tablet_manager.cc:1428] T 585b46c4b8d24ad89dd4d068eba5c9f8 P 66412a87e1fe45ff8f196b0684154ded: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260619 14:46:36.527899 2729 tablet_bootstrap.cc:492] T 585b46c4b8d24ad89dd4d068eba5c9f8 P 8bf8f0ae0321449c8d16ca2db42ed2f7: No bootstrap required, opened a new log
I20260619 14:46:36.528019 2729 ts_tablet_manager.cc:1397] T 585b46c4b8d24ad89dd4d068eba5c9f8 P 8bf8f0ae0321449c8d16ca2db42ed2f7: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260619 14:46:36.528134 2727 tablet_bootstrap.cc:492] T 585b46c4b8d24ad89dd4d068eba5c9f8 P 2339236c64244c94929ddd818b718bfd: No bootstrap required, opened a new log
I20260619 14:46:36.528252 2727 ts_tablet_manager.cc:1397] T 585b46c4b8d24ad89dd4d068eba5c9f8 P 2339236c64244c94929ddd818b718bfd: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 14:46:36.528524 2729 raft_consensus.cc:348] T 585b46c4b8d24ad89dd4d068eba5c9f8 P 8bf8f0ae0321449c8d16ca2db42ed2f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66412a87e1fe45ff8f196b0684154ded" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41829 } } peers { permanent_uuid: "2339236c64244c94929ddd818b718bfd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39357 } } peers { permanent_uuid: "8bf8f0ae0321449c8d16ca2db42ed2f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34533 } }
I20260619 14:46:36.528668 2729 raft_consensus.cc:374] T 585b46c4b8d24ad89dd4d068eba5c9f8 P 8bf8f0ae0321449c8d16ca2db42ed2f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:46:36.528724 2729 raft_consensus.cc:729] T 585b46c4b8d24ad89dd4d068eba5c9f8 P 8bf8f0ae0321449c8d16ca2db42ed2f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bf8f0ae0321449c8d16ca2db42ed2f7, State: Initialized, Role: FOLLOWER
I20260619 14:46:36.528628 2727 raft_consensus.cc:348] T 585b46c4b8d24ad89dd4d068eba5c9f8 P 2339236c64244c94929ddd818b718bfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66412a87e1fe45ff8f196b0684154ded" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41829 } } peers { permanent_uuid: "2339236c64244c94929ddd818b718bfd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39357 } } peers { permanent_uuid: "8bf8f0ae0321449c8d16ca2db42ed2f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34533 } }
I20260619 14:46:36.528805 2727 raft_consensus.cc:374] T 585b46c4b8d24ad89dd4d068eba5c9f8 P 2339236c64244c94929ddd818b718bfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:46:36.528851 2727 raft_consensus.cc:729] T 585b46c4b8d24ad89dd4d068eba5c9f8 P 2339236c64244c94929ddd818b718bfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2339236c64244c94929ddd818b718bfd, State: Initialized, Role: FOLLOWER
I20260619 14:46:36.528857 2729 consensus_queue.cc:260] T 585b46c4b8d24ad89dd4d068eba5c9f8 P 8bf8f0ae0321449c8d16ca2db42ed2f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66412a87e1fe45ff8f196b0684154ded" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41829 } } peers { permanent_uuid: "2339236c64244c94929ddd818b718bfd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39357 } } peers { permanent_uuid: "8bf8f0ae0321449c8d16ca2db42ed2f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34533 } }
I20260619 14:46:36.528985 2727 consensus_queue.cc:260] T 585b46c4b8d24ad89dd4d068eba5c9f8 P 2339236c64244c94929ddd818b718bfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66412a87e1fe45ff8f196b0684154ded" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41829 } } peers { permanent_uuid: "2339236c64244c94929ddd818b718bfd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39357 } } peers { permanent_uuid: "8bf8f0ae0321449c8d16ca2db42ed2f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34533 } }
I20260619 14:46:36.529160 2729 ts_tablet_manager.cc:1428] T 585b46c4b8d24ad89dd4d068eba5c9f8 P 8bf8f0ae0321449c8d16ca2db42ed2f7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:46:36.529702 2727 ts_tablet_manager.cc:1428] T 585b46c4b8d24ad89dd4d068eba5c9f8 P 2339236c64244c94929ddd818b718bfd: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260619 14:46:36.689991 2642 raft_consensus.cc:482] T 585b46c4b8d24ad89dd4d068eba5c9f8 P 2339236c64244c94929ddd818b718bfd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:46:36.690189 2642 raft_consensus.cc:504] T 585b46c4b8d24ad89dd4d068eba5c9f8 P 2339236c64244c94929ddd818b718bfd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66412a87e1fe45ff8f196b0684154ded" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41829 } } peers { permanent_uuid: "2339236c64244c94929ddd818b718bfd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39357 } } peers { permanent_uuid: "8bf8f0ae0321449c8d16ca2db42ed2f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34533 } }
I20260619 14:46:36.690675 2642 leader_election.cc:288] T 585b46c4b8d24ad89dd4d068eba5c9f8 P 2339236c64244c94929ddd818b718bfd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 66412a87e1fe45ff8f196b0684154ded (127.0.98.1:41829), 8bf8f0ae0321449c8d16ca2db42ed2f7 (127.0.98.3:34533)
I20260619 14:46:36.691184 2285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "585b46c4b8d24ad89dd4d068eba5c9f8" candidate_uuid: "2339236c64244c94929ddd818b718bfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66412a87e1fe45ff8f196b0684154ded" is_pre_election: true
I20260619 14:46:36.691203 2564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "585b46c4b8d24ad89dd4d068eba5c9f8" candidate_uuid: "2339236c64244c94929ddd818b718bfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bf8f0ae0321449c8d16ca2db42ed2f7" is_pre_election: true
I20260619 14:46:36.691352 2564 raft_consensus.cc:2454] T 585b46c4b8d24ad89dd4d068eba5c9f8 P 8bf8f0ae0321449c8d16ca2db42ed2f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2339236c64244c94929ddd818b718bfd in term 0.
I20260619 14:46:36.691365 2285 raft_consensus.cc:2454] T 585b46c4b8d24ad89dd4d068eba5c9f8 P 66412a87e1fe45ff8f196b0684154ded [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2339236c64244c94929ddd818b718bfd in term 0.
I20260619 14:46:36.691707 2356 leader_election.cc:302] T 585b46c4b8d24ad89dd4d068eba5c9f8 P 2339236c64244c94929ddd818b718bfd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2339236c64244c94929ddd818b718bfd, 8bf8f0ae0321449c8d16ca2db42ed2f7; no voters:
I20260619 14:46:36.691921 2642 raft_consensus.cc:2781] T 585b46c4b8d24ad89dd4d068eba5c9f8 P 2339236c64244c94929ddd818b718bfd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:46:36.692004 2642 raft_consensus.cc:482] T 585b46c4b8d24ad89dd4d068eba5c9f8 P 2339236c64244c94929ddd818b718bfd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:46:36.692062 2642 raft_consensus.cc:3037] T 585b46c4b8d24ad89dd4d068eba5c9f8 P 2339236c64244c94929ddd818b718bfd [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:46:36.693184 2642 raft_consensus.cc:504] T 585b46c4b8d24ad89dd4d068eba5c9f8 P 2339236c64244c94929ddd818b718bfd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66412a87e1fe45ff8f196b0684154ded" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41829 } } peers { permanent_uuid: "2339236c64244c94929ddd818b718bfd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39357 } } peers { permanent_uuid: "8bf8f0ae0321449c8d16ca2db42ed2f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34533 } }
I20260619 14:46:36.693578 2642 leader_election.cc:288] T 585b46c4b8d24ad89dd4d068eba5c9f8 P 2339236c64244c94929ddd818b718bfd [CANDIDATE]: Term 1 election: Requested vote from peers 66412a87e1fe45ff8f196b0684154ded (127.0.98.1:41829), 8bf8f0ae0321449c8d16ca2db42ed2f7 (127.0.98.3:34533)
I20260619 14:46:36.693984 2564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "585b46c4b8d24ad89dd4d068eba5c9f8" candidate_uuid: "2339236c64244c94929ddd818b718bfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8bf8f0ae0321449c8d16ca2db42ed2f7"
I20260619 14:46:36.694041 2285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "585b46c4b8d24ad89dd4d068eba5c9f8" candidate_uuid: "2339236c64244c94929ddd818b718bfd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66412a87e1fe45ff8f196b0684154ded"
I20260619 14:46:36.694129 2564 raft_consensus.cc:3037] T 585b46c4b8d24ad89dd4d068eba5c9f8 P 8bf8f0ae0321449c8d16ca2db42ed2f7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:46:36.694191 2285 raft_consensus.cc:3037] T 585b46c4b8d24ad89dd4d068eba5c9f8 P 66412a87e1fe45ff8f196b0684154ded [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:46:36.695120 2564 raft_consensus.cc:2454] T 585b46c4b8d24ad89dd4d068eba5c9f8 P 8bf8f0ae0321449c8d16ca2db42ed2f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2339236c64244c94929ddd818b718bfd in term 1.
I20260619 14:46:36.695174 2285 raft_consensus.cc:2454] T 585b46c4b8d24ad89dd4d068eba5c9f8 P 66412a87e1fe45ff8f196b0684154ded [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2339236c64244c94929ddd818b718bfd in term 1.
I20260619 14:46:36.695456 2356 leader_election.cc:302] T 585b46c4b8d24ad89dd4d068eba5c9f8 P 2339236c64244c94929ddd818b718bfd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2339236c64244c94929ddd818b718bfd, 8bf8f0ae0321449c8d16ca2db42ed2f7; no voters:
I20260619 14:46:36.695650 2642 raft_consensus.cc:2781] T 585b46c4b8d24ad89dd4d068eba5c9f8 P 2339236c64244c94929ddd818b718bfd [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:46:36.695744 2642 raft_consensus.cc:686] T 585b46c4b8d24ad89dd4d068eba5c9f8 P 2339236c64244c94929ddd818b718bfd [term 1 LEADER]: Becoming Leader. State: Replica: 2339236c64244c94929ddd818b718bfd, State: Running, Role: LEADER
I20260619 14:46:36.695883 2642 consensus_queue.cc:237] T 585b46c4b8d24ad89dd4d068eba5c9f8 P 2339236c64244c94929ddd818b718bfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66412a87e1fe45ff8f196b0684154ded" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41829 } } peers { permanent_uuid: "2339236c64244c94929ddd818b718bfd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39357 } } peers { permanent_uuid: "8bf8f0ae0321449c8d16ca2db42ed2f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34533 } }
I20260619 14:46:36.697957 2126 catalog_manager.cc:5495] T 585b46c4b8d24ad89dd4d068eba5c9f8 P 2339236c64244c94929ddd818b718bfd reported cstate change: term changed from 0 to 1, leader changed from <none> to 2339236c64244c94929ddd818b718bfd (127.0.98.2). New cstate: current_term: 1 leader_uuid: "2339236c64244c94929ddd818b718bfd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66412a87e1fe45ff8f196b0684154ded" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2339236c64244c94929ddd818b718bfd" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8bf8f0ae0321449c8d16ca2db42ed2f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34533 } health_report { overall_health: UNKNOWN } } }
I20260619 14:46:36.734675 2285 raft_consensus.cc:1261] T 585b46c4b8d24ad89dd4d068eba5c9f8 P 66412a87e1fe45ff8f196b0684154ded [term 1 FOLLOWER]: Refusing update from remote peer 2339236c64244c94929ddd818b718bfd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:46:36.734747 2564 raft_consensus.cc:1261] T 585b46c4b8d24ad89dd4d068eba5c9f8 P 8bf8f0ae0321449c8d16ca2db42ed2f7 [term 1 FOLLOWER]: Refusing update from remote peer 2339236c64244c94929ddd818b718bfd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:46:36.735168 2702 consensus_queue.cc:1035] T 585b46c4b8d24ad89dd4d068eba5c9f8 P 2339236c64244c94929ddd818b718bfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "66412a87e1fe45ff8f196b0684154ded" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:46:36.735345 2697 consensus_queue.cc:1035] T 585b46c4b8d24ad89dd4d068eba5c9f8 P 2339236c64244c94929ddd818b718bfd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8bf8f0ae0321449c8d16ca2db42ed2f7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 14:46:36.750799 2695 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260619 14:46:38.309005 2128 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33944:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:46:38.309211 2128 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33944:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:46:38.312270 2128 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 13d2cf43b72142a48e518f2725623771: Sending DeleteTablet for 3 replicas of tablet 585b46c4b8d24ad89dd4d068eba5c9f8
I20260619 14:46:38.313030 2265 tablet_service.cc:1514] Processing DeleteTablet for tablet 585b46c4b8d24ad89dd4d068eba5c9f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:46:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35320
I20260619 14:46:38.313088 2404 tablet_service.cc:1514] Processing DeleteTablet for tablet 585b46c4b8d24ad89dd4d068eba5c9f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:46:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47526
I20260619 14:46:38.313370 2545 tablet_service.cc:1514] Processing DeleteTablet for tablet 585b46c4b8d24ad89dd4d068eba5c9f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:46:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51396
I20260619 14:46:38.313455 2765 tablet_replica.cc:330] T 585b46c4b8d24ad89dd4d068eba5c9f8 P 66412a87e1fe45ff8f196b0684154ded: stopping tablet replica
I20260619 14:46:38.313609 2765 raft_consensus.cc:2229] T 585b46c4b8d24ad89dd4d068eba5c9f8 P 66412a87e1fe45ff8f196b0684154ded [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:46:38.313771 2765 raft_consensus.cc:2258] T 585b46c4b8d24ad89dd4d068eba5c9f8 P 66412a87e1fe45ff8f196b0684154ded [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:46:38.313791 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 2200
I20260619 14:46:38.313915 2767 tablet_replica.cc:330] T 585b46c4b8d24ad89dd4d068eba5c9f8 P 8bf8f0ae0321449c8d16ca2db42ed2f7: stopping tablet replica
I20260619 14:46:38.314025 2767 raft_consensus.cc:2229] T 585b46c4b8d24ad89dd4d068eba5c9f8 P 8bf8f0ae0321449c8d16ca2db42ed2f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:46:38.314702 2767 raft_consensus.cc:2258] T 585b46c4b8d24ad89dd4d068eba5c9f8 P 8bf8f0ae0321449c8d16ca2db42ed2f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:46:38.315431 2767 ts_tablet_manager.cc:1905] T 585b46c4b8d24ad89dd4d068eba5c9f8 P 8bf8f0ae0321449c8d16ca2db42ed2f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:46:38.318105 2767 ts_tablet_manager.cc:1918] T 585b46c4b8d24ad89dd4d068eba5c9f8 P 8bf8f0ae0321449c8d16ca2db42ed2f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 14:46:38.318230 2767 log.cc:1196] T 585b46c4b8d24ad89dd4d068eba5c9f8 P 8bf8f0ae0321449c8d16ca2db42ed2f7: Deleting WAL directory at /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/minicluster-data/ts-2/wal/wals/585b46c4b8d24ad89dd4d068eba5c9f8
I20260619 14:46:38.318465 2766 tablet_replica.cc:330] T 585b46c4b8d24ad89dd4d068eba5c9f8 P 2339236c64244c94929ddd818b718bfd: stopping tablet replica
I20260619 14:46:38.318578 2767 ts_tablet_manager.cc:1939] T 585b46c4b8d24ad89dd4d068eba5c9f8 P 8bf8f0ae0321449c8d16ca2db42ed2f7: Deleting consensus metadata
I20260619 14:46:38.318670 2766 raft_consensus.cc:2229] T 585b46c4b8d24ad89dd4d068eba5c9f8 P 2339236c64244c94929ddd818b718bfd [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:46:38.318936 2766 raft_consensus.cc:2258] T 585b46c4b8d24ad89dd4d068eba5c9f8 P 2339236c64244c94929ddd818b718bfd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:46:38.319507 2112 catalog_manager.cc:4841] TS 8bf8f0ae0321449c8d16ca2db42ed2f7 (127.0.98.3:34533): tablet 585b46c4b8d24ad89dd4d068eba5c9f8 (table test-table [id=3b92b399f3cc4252b6bc738b3638212d]) successfully deleted
I20260619 14:46:38.319767 2766 ts_tablet_manager.cc:1905] T 585b46c4b8d24ad89dd4d068eba5c9f8 P 2339236c64244c94929ddd818b718bfd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:46:38.322714 2766 ts_tablet_manager.cc:1918] T 585b46c4b8d24ad89dd4d068eba5c9f8 P 2339236c64244c94929ddd818b718bfd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260619 14:46:38.322852 2766 log.cc:1196] T 585b46c4b8d24ad89dd4d068eba5c9f8 P 2339236c64244c94929ddd818b718bfd: Deleting WAL directory at /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781880383132139-392-0/minicluster-data/ts-1/wal/wals/585b46c4b8d24ad89dd4d068eba5c9f8
W20260619 14:46:38.322844 2114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41829 (error 108)
I20260619 14:46:38.323107 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 2340
I20260619 14:46:38.329850 2769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:38.329059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41829 (local address 127.0.0.1:35338)
0619 14:46:38.329565 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:38.329681 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":247,"threads_started":1}
W20260619 14:46:38.330091 2114 catalog_manager.cc:4584] TS 66412a87e1fe45ff8f196b0684154ded (127.0.98.1:41829): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 585b46c4b8d24ad89dd4d068eba5c9f8: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41829: connect: Connection refused (error 111)
I20260619 14:46:38.330895 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 2480
I20260619 14:46:38.331211 2769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:38.330897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39357 (local address 127.0.0.1:47564)
0619 14:46:38.331054 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:38.331116 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:39357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260619 14:46:38.337674 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 2096
2026-06-19T14:46:38Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (7007 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880383132139-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:46:38 dist-test-slave-2sb1 krb5kdc[2772](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:46:38 dist-test-slave-2sb1 krb5kdc[2772](info): set up 2 sockets
Jun 19 14:46:38 dist-test-slave-2sb1 krb5kdc[2772](info): commencing operation
krb5kdc: starting...
W20260619 14:46:39.476929 392 mini_kdc.cc:121] Time spent starting KDC: real 1.114s user 0.006s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880383132139-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880383132139-392-0/krb5kdc/test-user.keytab.
Jun 19 14:46:39 dist-test-slave-2sb1 krb5kdc[2772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:46:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:46:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 14:46:39.621694 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880383132139-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880383132139-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880383132139-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39659
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39659
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:46:39.748301 2785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:46:39.748713 2785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:46:39.748845 2785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:46:39.753592 2785 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:46:39.753736 2785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:46:39.753777 2785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:46:39.753928 2785 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:46:39.754025 2785 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:46:39.760049 2785 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880383132139-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880383132139-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39659
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39659
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2785
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880383132139-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:46:39.761313 2785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:46:39.762404 2785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:46:39.771842 2790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:39.771719 2791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:39.771955 2793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:39.772703 2785 server_base.cc:1034] running on GCE node
I20260619 14:46:39.773449 2785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:46:39.774887 2785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:46:39.776118 2785 hybrid_clock.cc:648] HybridClock initialized: now 1781880399776079 us; error 57 us; skew 500 ppm
Jun 19 14:46:39 dist-test-slave-2sb1 krb5kdc[2772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880399, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:46:39.779822 2785 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 14:46:39.781035 2785 webserver.cc:460] Webserver started at http://127.0.98.62:43087/ using document root <none> and password file <none>
I20260619 14:46:39.781698 2785 fs_manager.cc:359] Metadata directory not provided
I20260619 14:46:39.781812 2785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:46:39.782097 2785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:46:39.784934 2785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880383132139-392-0/minicluster-data/master-0/data/instance:
uuid: "01b91cf2152a402ca629e7b1ccc7b4f9"
format_stamp: "Formatted at 2026-06-19 14:46:39 on dist-test-slave-2sb1"
server_key: "13929b641c9e029b6e87660db9f0b58e"
server_key_iv: "026036d5e1d44562ff6708ec17ad4bfb"
server_key_version: "encryptionkey@0"
I20260619 14:46:39.786091 2785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880383132139-392-0/minicluster-data/master-0/wal/instance:
uuid: "01b91cf2152a402ca629e7b1ccc7b4f9"
format_stamp: "Formatted at 2026-06-19 14:46:39 on dist-test-slave-2sb1"
server_key: "13929b641c9e029b6e87660db9f0b58e"
server_key_iv: "026036d5e1d44562ff6708ec17ad4bfb"
server_key_version: "encryptionkey@0"
I20260619 14:46:39.790649 2785 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260619 14:46:39.793666 2800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:39.795578 2785 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 14:46:39.795738 2785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880383132139-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880383132139-392-0/minicluster-data/master-0/wal
uuid: "01b91cf2152a402ca629e7b1ccc7b4f9"
format_stamp: "Formatted at 2026-06-19 14:46:39 on dist-test-slave-2sb1"
server_key: "13929b641c9e029b6e87660db9f0b58e"
server_key_iv: "026036d5e1d44562ff6708ec17ad4bfb"
server_key_version: "encryptionkey@0"
I20260619 14:46:39.795948 2785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880383132139-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880383132139-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880383132139-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:46:39.818005 2785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:46:39.821856 2785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:46:39.822083 2785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:46:39.829926 2785 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39659
I20260619 14:46:39.829945 2852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39659 every 8 connection(s)
I20260619 14:46:39.830973 2785 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:46:39.833984 2853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:46:39.839802 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 2785
I20260619 14:46:39.839910 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880383132139-392-0/minicluster-data/master-0/wal/instance
I20260619 14:46:39.840219 392 external_mini_cluster.cc:1428] Setting key 39b8b14e36b428b144ad4c2793da9fa4
I20260619 14:46:39.840483 2853 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01b91cf2152a402ca629e7b1ccc7b4f9: Bootstrap starting.
I20260619 14:46:39.843564 2853 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 01b91cf2152a402ca629e7b1ccc7b4f9: Neither blocks nor log segments found. Creating new log.
I20260619 14:46:39.844456 2853 log.cc:824] T 00000000000000000000000000000000 P 01b91cf2152a402ca629e7b1ccc7b4f9: Log is configured to *not* fsync() on all Append() calls
I20260619 14:46:39.846845 2853 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01b91cf2152a402ca629e7b1ccc7b4f9: No bootstrap required, opened a new log
Jun 19 14:46:39 dist-test-slave-2sb1 krb5kdc[2772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:46:39.851804 2853 raft_consensus.cc:348] T 00000000000000000000000000000000 P 01b91cf2152a402ca629e7b1ccc7b4f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01b91cf2152a402ca629e7b1ccc7b4f9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39659 } }
I20260619 14:46:39.852114 2853 raft_consensus.cc:374] T 00000000000000000000000000000000 P 01b91cf2152a402ca629e7b1ccc7b4f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:46:39.852157 2853 raft_consensus.cc:729] T 00000000000000000000000000000000 P 01b91cf2152a402ca629e7b1ccc7b4f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01b91cf2152a402ca629e7b1ccc7b4f9, State: Initialized, Role: FOLLOWER
I20260619 14:46:39.852995 2853 consensus_queue.cc:260] T 00000000000000000000000000000000 P 01b91cf2152a402ca629e7b1ccc7b4f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01b91cf2152a402ca629e7b1ccc7b4f9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39659 } }
I20260619 14:46:39.853168 2853 raft_consensus.cc:388] T 00000000000000000000000000000000 P 01b91cf2152a402ca629e7b1ccc7b4f9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:46:39.853284 2853 raft_consensus.cc:482] T 00000000000000000000000000000000 P 01b91cf2152a402ca629e7b1ccc7b4f9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:46:39.853433 2853 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 01b91cf2152a402ca629e7b1ccc7b4f9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:46:39.854702 2853 raft_consensus.cc:504] T 00000000000000000000000000000000 P 01b91cf2152a402ca629e7b1ccc7b4f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01b91cf2152a402ca629e7b1ccc7b4f9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39659 } }
I20260619 14:46:39.855101 2853 leader_election.cc:302] T 00000000000000000000000000000000 P 01b91cf2152a402ca629e7b1ccc7b4f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 01b91cf2152a402ca629e7b1ccc7b4f9; no voters:
I20260619 14:46:39.855502 2853 leader_election.cc:288] T 00000000000000000000000000000000 P 01b91cf2152a402ca629e7b1ccc7b4f9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:46:39.855639 2858 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 01b91cf2152a402ca629e7b1ccc7b4f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:46:39.855840 2858 raft_consensus.cc:686] T 00000000000000000000000000000000 P 01b91cf2152a402ca629e7b1ccc7b4f9 [term 1 LEADER]: Becoming Leader. State: Replica: 01b91cf2152a402ca629e7b1ccc7b4f9, State: Running, Role: LEADER
I20260619 14:46:39.856180 2858 consensus_queue.cc:237] T 00000000000000000000000000000000 P 01b91cf2152a402ca629e7b1ccc7b4f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01b91cf2152a402ca629e7b1ccc7b4f9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39659 } }
I20260619 14:46:39.856954 2853 sys_catalog.cc:564] T 00000000000000000000000000000000 P 01b91cf2152a402ca629e7b1ccc7b4f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:46:39.857916 2859 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01b91cf2152a402ca629e7b1ccc7b4f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "01b91cf2152a402ca629e7b1ccc7b4f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01b91cf2152a402ca629e7b1ccc7b4f9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39659 } } }
I20260619 14:46:39.858065 2859 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01b91cf2152a402ca629e7b1ccc7b4f9 [sys.catalog]: This master's current role is: LEADER
I20260619 14:46:39.858407 2860 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01b91cf2152a402ca629e7b1ccc7b4f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 01b91cf2152a402ca629e7b1ccc7b4f9. Latest consensus state: current_term: 1 leader_uuid: "01b91cf2152a402ca629e7b1ccc7b4f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01b91cf2152a402ca629e7b1ccc7b4f9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39659 } } }
I20260619 14:46:39.858506 2860 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01b91cf2152a402ca629e7b1ccc7b4f9 [sys.catalog]: This master's current role is: LEADER
I20260619 14:46:39.858999 2867 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:46:39.860575 2856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:39.841853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37590 (local address 127.0.98.62:39659)
0619 14:46:39.842436 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:39.842453 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:39.842495 (+ 42us) server_negotiation.cc:407] Connection header received
0619 14:46:39.843505 (+ 1010us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:39.843529 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:39.844053 (+ 524us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:39.844478 (+ 425us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:39.845553 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:39.846675 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:39.847740 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:39.848312 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:39.848515 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:39.852829 (+ 4314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:39.852836 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:39.852853 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:39.852894 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:39.855423 (+ 2529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:39.856131 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:39.856138 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:39.856144 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:39.856325 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:39.856700 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:39.856704 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:39.856706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:39.857633 (+ 927us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:39.858031 (+ 398us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:39.858901 (+ 870us) server_negotiation.cc:299] Negotiation successful
0619 14:46:39.859420 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":186,"threads_started":1}
I20260619 14:46:39.862329 2867 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:46:39.868749 2867 catalog_manager.cc:1347] Generated new cluster ID: 3d4e5d935368433c82594ff62564e30f
I20260619 14:46:39.868929 2867 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:46:39.885838 2867 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:46:39.886953 2867 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:46:39.891980 2867 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 01b91cf2152a402ca629e7b1ccc7b4f9: Generated new TSK 0
I20260619 14:46:39.893326 2867 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880383132139-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880383132139-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 14:46:39.930776 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880383132139-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880383132139-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880383132139-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880383132139-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39659
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880383132139-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:46:40.068409 2879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:46:40.068801 2879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:46:40.068856 2879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:46:40.074223 2879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:46:40.074328 2879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:46:40.074395 2879 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 14:46:40.081434 2879 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880383132139-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880383132139-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880383132139-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880383132139-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39659
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2879
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880383132139-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:46:40.082782 2879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:46:40.083994 2879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:46:40.096378 2884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:40.096558 2885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:40.096961 2887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:40.096953 2879 server_base.cc:1034] running on GCE node
I20260619 14:46:40.097556 2879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:46:40.098279 2879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:46:40.099486 2879 hybrid_clock.cc:648] HybridClock initialized: now 1781880400099427 us; error 73 us; skew 500 ppm
Jun 19 14:46:40 dist-test-slave-2sb1 krb5kdc[2772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880400, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:46:40.103549 2879 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 14:46:40.104995 2879 webserver.cc:460] Webserver started at http://127.0.98.1:35395/ using document root <none> and password file <none>
I20260619 14:46:40.105646 2879 fs_manager.cc:359] Metadata directory not provided
I20260619 14:46:40.105717 2879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:46:40.105959 2879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:46:40.108361 2879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880383132139-392-0/minicluster-data/ts-0/data/instance:
uuid: "9d5ac468ca3e4490a5f189243b53dd71"
format_stamp: "Formatted at 2026-06-19 14:46:40 on dist-test-slave-2sb1"
server_key: "58886564813f47a2fe432c2af5474bd6"
server_key_iv: "9dffb62b41533380f4d1be3c033b8cd2"
server_key_version: "encryptionkey@0"
I20260619 14:46:40.109082 2879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880383132139-392-0/minicluster-data/ts-0/wal/instance:
uuid: "9d5ac468ca3e4490a5f189243b53dd71"
format_stamp: "Formatted at 2026-06-19 14:46:40 on dist-test-slave-2sb1"
server_key: "58886564813f47a2fe432c2af5474bd6"
server_key_iv: "9dffb62b41533380f4d1be3c033b8cd2"
server_key_version: "encryptionkey@0"
I20260619 14:46:40.114184 2879 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260619 14:46:40.118618 2894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:40.120455 2879 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260619 14:46:40.120710 2879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880383132139-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880383132139-392-0/minicluster-data/ts-0/wal
uuid: "9d5ac468ca3e4490a5f189243b53dd71"
format_stamp: "Formatted at 2026-06-19 14:46:40 on dist-test-slave-2sb1"
server_key: "58886564813f47a2fe432c2af5474bd6"
server_key_iv: "9dffb62b41533380f4d1be3c033b8cd2"
server_key_version: "encryptionkey@0"
I20260619 14:46:40.120873 2879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880383132139-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880383132139-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880383132139-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:46:40.150246 2879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:46:40.155763 2879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:46:40.156080 2879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:46:40.156953 2879 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:46:40.158041 2879 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:46:40.158121 2879 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:40.158236 2879 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:46:40.158295 2879 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:40.170852 2879 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36633
I20260619 14:46:40.170898 3007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36633 every 8 connection(s)
I20260619 14:46:40.172029 2879 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880383132139-392-0/minicluster-data/ts-0/data/info.pb
I20260619 14:46:40.179911 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 2879
I20260619 14:46:40.180073 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880383132139-392-0/minicluster-data/ts-0/wal/instance
I20260619 14:46:40.180382 392 external_mini_cluster.cc:1428] Setting key 72a24f4eab156d88d4690600df6d61fc
Jun 19 14:46:40 dist-test-slave-2sb1 krb5kdc[2772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880400, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:46:40.192569 2856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:40.174653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56259 (local address 127.0.98.62:39659)
0619 14:46:40.174925 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:40.174930 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:40.176032 (+ 1102us) server_negotiation.cc:407] Connection header received
0619 14:46:40.177813 (+ 1781us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:40.177817 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:40.177875 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:40.177968 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:40.179726 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:40.180502 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:40.182413 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:40.182703 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:40.182851 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:40.186706 (+ 3855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:40.186715 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:40.186722 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:40.186763 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:40.188962 (+ 2199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:40.189934 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:40.189938 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:40.189940 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:40.190041 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:40.190587 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:40.190590 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:40.190592 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:40.190905 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:40.191014 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:40.192293 (+ 1279us) server_negotiation.cc:299] Negotiation successful
0619 14:46:40.192435 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 14:46:40.194296 3010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:40.175277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39659 (local address 127.0.98.1:56259)
0619 14:46:40.175854 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:40.175905 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:40.177499 (+ 1594us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:40.178238 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:40.178247 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:40.178709 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:46:40.179461 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:40.179480 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:40.180706 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:40.180711 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:40.182108 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:40.182127 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:40.182941 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:40.182949 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:40.183241 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:40.184052 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:46:40.184081 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:46:40.186424 (+ 2343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:46:40.189141 (+ 2717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:40.189154 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:40.189182 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:40.189701 (+ 519us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:40.190165 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:40.190169 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:40.190172 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:40.190430 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:40.191166 (+ 736us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:46:40.191177 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:46:40.191601 (+ 424us) client_negotiation.cc:769] Sending connection context
0619 14:46:40.192192 (+ 591us) client_negotiation.cc:240] Negotiation successful
0619 14:46:40.192640 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":146,"threads_started":1}
I20260619 14:46:40.196462 3008 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39659
I20260619 14:46:40.196846 3008 heartbeater.cc:464] Registering TS with master...
I20260619 14:46:40.197528 3008 heartbeater.cc:511] Master 127.0.98.62:39659 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 14:46:40.199383 2817 ts_manager.cc:194] Registered new tserver with Master: 9d5ac468ca3e4490a5f189243b53dd71 (127.0.98.1:36633)
I20260619 14:46:40.200940 2817 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:56259
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880383132139-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880383132139-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 14:46:40.223016 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880383132139-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880383132139-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880383132139-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880383132139-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39659
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880383132139-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:46:40.346954 3013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:46:40.347306 3013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:46:40.347385 3013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:46:40.351794 3013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:46:40.351920 3013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:46:40.352023 3013 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 14:46:40.359166 3013 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880383132139-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880383132139-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880383132139-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880383132139-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39659
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3013
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880383132139-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:46:40.360374 3013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:46:40.361959 3013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:46:40.373731 3019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:40.373744 3021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:40.373914 3013 server_base.cc:1034] running on GCE node
W20260619 14:46:40.373811 3018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:40.374557 3013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:46:40.375263 3013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:46:40.376489 3013 hybrid_clock.cc:648] HybridClock initialized: now 1781880400376458 us; error 39 us; skew 500 ppm
Jun 19 14:46:40 dist-test-slave-2sb1 krb5kdc[2772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880400, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:46:40.380137 3013 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 14:46:40.381415 3013 webserver.cc:460] Webserver started at http://127.0.98.2:34775/ using document root <none> and password file <none>
I20260619 14:46:40.382082 3013 fs_manager.cc:359] Metadata directory not provided
I20260619 14:46:40.382155 3013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:46:40.382378 3013 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:46:40.384604 3013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880383132139-392-0/minicluster-data/ts-1/data/instance:
uuid: "d2729b5dbdda4e55afb48e7d754b9bc4"
format_stamp: "Formatted at 2026-06-19 14:46:40 on dist-test-slave-2sb1"
server_key: "ae657bc0a8a1faf838dcb58e3236d22c"
server_key_iv: "19fa6b2aedbd17ac30535e8491620407"
server_key_version: "encryptionkey@0"
I20260619 14:46:40.385308 3013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880383132139-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d2729b5dbdda4e55afb48e7d754b9bc4"
format_stamp: "Formatted at 2026-06-19 14:46:40 on dist-test-slave-2sb1"
server_key: "ae657bc0a8a1faf838dcb58e3236d22c"
server_key_iv: "19fa6b2aedbd17ac30535e8491620407"
server_key_version: "encryptionkey@0"
I20260619 14:46:40.389627 3013 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 14:46:40.393473 3028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:40.395263 3013 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260619 14:46:40.395409 3013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880383132139-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880383132139-392-0/minicluster-data/ts-1/wal
uuid: "d2729b5dbdda4e55afb48e7d754b9bc4"
format_stamp: "Formatted at 2026-06-19 14:46:40 on dist-test-slave-2sb1"
server_key: "ae657bc0a8a1faf838dcb58e3236d22c"
server_key_iv: "19fa6b2aedbd17ac30535e8491620407"
server_key_version: "encryptionkey@0"
I20260619 14:46:40.395673 3013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880383132139-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880383132139-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880383132139-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:46:40.413058 3013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:46:40.417562 3013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:46:40.417826 3013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:46:40.418488 3013 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:46:40.419428 3013 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:46:40.419478 3013 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:40.419548 3013 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:46:40.419581 3013 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:40.431955 3013 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41095
I20260619 14:46:40.431982 3141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41095 every 8 connection(s)
I20260619 14:46:40.433408 3013 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880383132139-392-0/minicluster-data/ts-1/data/info.pb
I20260619 14:46:40.441888 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 3013
I20260619 14:46:40.441998 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880383132139-392-0/minicluster-data/ts-1/wal/instance
I20260619 14:46:40.442421 392 external_mini_cluster.cc:1428] Setting key 844f51ea828bd0d212f69fa4181cf806
Jun 19 14:46:40 dist-test-slave-2sb1 krb5kdc[2772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880400, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:46:40.453634 2856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:40.435677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37167 (local address 127.0.98.62:39659)
0619 14:46:40.435971 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:40.435977 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:40.437103 (+ 1126us) server_negotiation.cc:407] Connection header received
0619 14:46:40.438732 (+ 1629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:40.438736 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:40.438814 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:40.438905 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:40.440585 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:40.441334 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:40.442822 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:40.443110 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:40.443221 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:40.447548 (+ 4327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:40.447552 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:40.447554 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:40.447581 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:40.449955 (+ 2374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:40.451057 (+ 1102us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:40.451065 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:40.451070 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:40.451189 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:40.451732 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:40.451738 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:40.451743 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:40.452032 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:40.452159 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:40.453178 (+ 1019us) server_negotiation.cc:299] Negotiation successful
0619 14:46:40.453367 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188}
I20260619 14:46:40.454546 3144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:40.436204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39659 (local address 127.0.98.2:37167)
0619 14:46:40.436927 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:40.436978 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:40.438455 (+ 1477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:40.439176 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:40.439185 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:40.439842 (+ 657us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:46:40.440384 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:40.440401 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:40.441499 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:40.441502 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:40.442678 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:40.442689 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:40.443314 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:40.443319 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:40.443548 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:40.444511 (+ 963us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:46:40.444530 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:46:40.447252 (+ 2722us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:46:40.450484 (+ 3232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:40.450491 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:40.450508 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:40.450901 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:40.451403 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:40.451406 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:40.451408 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:40.451594 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:40.452197 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:46:40.452204 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:46:40.452735 (+ 531us) client_negotiation.cc:769] Sending connection context
0619 14:46:40.453091 (+ 356us) client_negotiation.cc:240] Negotiation successful
0619 14:46:40.453554 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":162,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 14:46:40.457044 3142 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39659
I20260619 14:46:40.457513 3142 heartbeater.cc:464] Registering TS with master...
I20260619 14:46:40.458364 3142 heartbeater.cc:511] Master 127.0.98.62:39659 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260619 14:46:40.460130 2817 ts_manager.cc:194] Registered new tserver with Master: d2729b5dbdda4e55afb48e7d754b9bc4 (127.0.98.2:41095)
I20260619 14:46:40.461797 2817 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:37167
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880383132139-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880383132139-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 14:46:40.481918 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880383132139-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880383132139-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880383132139-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880383132139-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39659
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880383132139-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:46:40.621259 3147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:46:40.621620 3147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:46:40.621675 3147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:46:40.626520 3147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:46:40.626610 3147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:46:40.626677 3147 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 14:46:40.632505 3147 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880383132139-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880383132139-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880383132139-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880383132139-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39659
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3147
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880383132139-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:46:40.633782 3147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:46:40.634876 3147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:46:40.645879 3152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:40.645902 3153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:40.645895 3155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:40.646417 3147 server_base.cc:1034] running on GCE node
I20260619 14:46:40.647049 3147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:46:40.647862 3147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:46:40.649071 3147 hybrid_clock.cc:648] HybridClock initialized: now 1781880400649056 us; error 33 us; skew 500 ppm
Jun 19 14:46:40 dist-test-slave-2sb1 krb5kdc[2772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880400, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:46:40.652464 3147 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 14:46:40.653622 3147 webserver.cc:460] Webserver started at http://127.0.98.3:43357/ using document root <none> and password file <none>
I20260619 14:46:40.654237 3147 fs_manager.cc:359] Metadata directory not provided
I20260619 14:46:40.654332 3147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:46:40.654570 3147 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:46:40.656870 3147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880383132139-392-0/minicluster-data/ts-2/data/instance:
uuid: "c8c7b0fa0a5f4608aa92a6fe537fa0f4"
format_stamp: "Formatted at 2026-06-19 14:46:40 on dist-test-slave-2sb1"
server_key: "03670a738f5fd1469103e62023413c50"
server_key_iv: "b24485c160c52006e75f4f441284ee0f"
server_key_version: "encryptionkey@0"
I20260619 14:46:40.657532 3147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880383132139-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c8c7b0fa0a5f4608aa92a6fe537fa0f4"
format_stamp: "Formatted at 2026-06-19 14:46:40 on dist-test-slave-2sb1"
server_key: "03670a738f5fd1469103e62023413c50"
server_key_iv: "b24485c160c52006e75f4f441284ee0f"
server_key_version: "encryptionkey@0"
I20260619 14:46:40.662043 3147 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260619 14:46:40.665398 3162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:40.667344 3147 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260619 14:46:40.667532 3147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880383132139-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880383132139-392-0/minicluster-data/ts-2/wal
uuid: "c8c7b0fa0a5f4608aa92a6fe537fa0f4"
format_stamp: "Formatted at 2026-06-19 14:46:40 on dist-test-slave-2sb1"
server_key: "03670a738f5fd1469103e62023413c50"
server_key_iv: "b24485c160c52006e75f4f441284ee0f"
server_key_version: "encryptionkey@0"
I20260619 14:46:40.667683 3147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880383132139-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880383132139-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880383132139-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:46:40.693351 3147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:46:40.697508 3147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:46:40.697742 3147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:46:40.698390 3147 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:46:40.699287 3147 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:46:40.699361 3147 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:40.699445 3147 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:46:40.699501 3147 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:40.711722 3147 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46229
I20260619 14:46:40.711738 3275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46229 every 8 connection(s)
I20260619 14:46:40.712743 3147 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880383132139-392-0/minicluster-data/ts-2/data/info.pb
I20260619 14:46:40.721364 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 3147
I20260619 14:46:40.721486 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781880383132139-392-0/minicluster-data/ts-2/wal/instance
I20260619 14:46:40.721845 392 external_mini_cluster.cc:1428] Setting key 294d2059a575fb6cbb29cc0a096b167a
Jun 19 14:46:40 dist-test-slave-2sb1 krb5kdc[2772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880400, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:46:40.730262 2856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:40.715387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41455 (local address 127.0.98.62:39659)
0619 14:46:40.715550 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:40.715554 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:40.716527 (+ 973us) server_negotiation.cc:407] Connection header received
0619 14:46:40.718116 (+ 1589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:40.718120 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:40.718173 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:40.718569 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:40.720032 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:40.720631 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:40.722596 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:40.722857 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:40.722991 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:40.726165 (+ 3174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:40.726169 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:40.726172 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:40.726201 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:40.727747 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:40.728525 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:40.728529 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:40.728530 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:40.728700 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:40.729111 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:40.729113 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:40.729114 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:40.729368 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:40.729459 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:40.730020 (+ 561us) server_negotiation.cc:299] Negotiation successful
0619 14:46:40.730139 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 14:46:40.731211 3278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:40.715736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39659 (local address 127.0.98.3:41455)
0619 14:46:40.716317 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:40.716370 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:40.717798 (+ 1428us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:40.718822 (+ 1024us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:40.718831 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:40.719265 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:46:40.719818 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:40.719834 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:40.720931 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:40.720935 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:40.722421 (+ 1486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:40.722436 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:40.723044 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:40.723049 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:40.723270 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:40.723971 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:46:40.723988 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:46:40.725877 (+ 1889us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:46:40.727895 (+ 2018us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:40.727901 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:40.727917 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:40.728216 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:40.728827 (+ 611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:40.728830 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:40.728831 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:40.729012 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:40.729495 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:46:40.729502 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:46:40.729785 (+ 283us) client_negotiation.cc:769] Sending connection context
0619 14:46:40.729964 (+ 179us) client_negotiation.cc:240] Negotiation successful
0619 14:46:40.730269 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":164,"threads_started":1}
I20260619 14:46:40.732749 3276 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39659
I20260619 14:46:40.733019 3276 heartbeater.cc:464] Registering TS with master...
I20260619 14:46:40.733518 3276 heartbeater.cc:511] Master 127.0.98.62:39659 requested a full tablet report, sending...
I20260619 14:46:40.734886 2817 ts_manager.cc:194] Registered new tserver with Master: c8c7b0fa0a5f4608aa92a6fe537fa0f4 (127.0.98.3:46229)
I20260619 14:46:40.736004 2817 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41455
I20260619 14:46:40.736951 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 14:46:40.751693 2856 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:46:40.749975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37600 (local address 127.0.98.62:39659)
0619 14:46:40.750118 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:40.750122 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:40.750387 (+ 265us) server_negotiation.cc:407] Connection header received
0619 14:46:40.750442 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:40.750444 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:40.750489 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:40.750572 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:40.751537 (+ 965us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37600: BlockingRecv error: recv got EOF from 127.0.0.1:37600 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 14:46:40.753043 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 2879
I20260619 14:46:40.759135 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 3013
I20260619 14:46:40.764559 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 3147
I20260619 14:46:40.770581 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 2785
2026-06-19T14:46:40Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2431 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781880383132139-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:46:40 dist-test-slave-2sb1 krb5kdc[3288](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:46:40 dist-test-slave-2sb1 krb5kdc[3288](info): set up 2 sockets
Jun 19 14:46:40 dist-test-slave-2sb1 krb5kdc[3288](info): commencing operation
krb5kdc: starting...
W20260619 14:46:41.933243 392 mini_kdc.cc:121] Time spent starting KDC: real 1.139s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781880383132139-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781880383132139-392-0/krb5kdc/test-user.keytab.
Jun 19 14:46:42 dist-test-slave-2sb1 krb5kdc[3288](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880402, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:46:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:46:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 14:46:42.085259 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781880383132139-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781880383132139-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781880383132139-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42201
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42201
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:46:42.213865 3301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:46:42.214174 3301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:46:42.214226 3301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:46:42.218499 3301 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:46:42.218587 3301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:46:42.218608 3301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:46:42.218626 3301 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:46:42.218641 3301 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:46:42.225085 3301 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781880383132139-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781880383132139-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42201
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3301
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781880383132139-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:46:42.226473 3301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:46:42.227662 3301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:46:42.237630 3309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:42.237629 3306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:42.237630 3307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:42.238504 3301 server_base.cc:1034] running on GCE node
I20260619 14:46:42.239256 3301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:46:42.240744 3301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:46:42.241986 3301 hybrid_clock.cc:648] HybridClock initialized: now 1781880402241953 us; error 52 us; skew 500 ppm
Jun 19 14:46:42 dist-test-slave-2sb1 krb5kdc[3288](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880402, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:46:42.245440 3301 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 14:46:42.246608 3301 webserver.cc:460] Webserver started at http://127.0.98.62:44425/ using document root <none> and password file <none>
I20260619 14:46:42.247236 3301 fs_manager.cc:359] Metadata directory not provided
I20260619 14:46:42.247335 3301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:46:42.247571 3301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:46:42.249806 3301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781880383132139-392-0/minicluster-data/master-0/data/instance:
uuid: "328fe308b68741828fc99ee88d906788"
format_stamp: "Formatted at 2026-06-19 14:46:42 on dist-test-slave-2sb1"
server_key: "8aa5e0db409c441e3a64a2c71aa4ad6f"
server_key_iv: "8b580c42bef7dddfba63ae208247df82"
server_key_version: "encryptionkey@0"
I20260619 14:46:42.250406 3301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781880383132139-392-0/minicluster-data/master-0/wal/instance:
uuid: "328fe308b68741828fc99ee88d906788"
format_stamp: "Formatted at 2026-06-19 14:46:42 on dist-test-slave-2sb1"
server_key: "8aa5e0db409c441e3a64a2c71aa4ad6f"
server_key_iv: "8b580c42bef7dddfba63ae208247df82"
server_key_version: "encryptionkey@0"
I20260619 14:46:42.254830 3301 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 14:46:42.257705 3316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:42.259295 3301 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:46:42.259558 3301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781880383132139-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781880383132139-392-0/minicluster-data/master-0/wal
uuid: "328fe308b68741828fc99ee88d906788"
format_stamp: "Formatted at 2026-06-19 14:46:42 on dist-test-slave-2sb1"
server_key: "8aa5e0db409c441e3a64a2c71aa4ad6f"
server_key_iv: "8b580c42bef7dddfba63ae208247df82"
server_key_version: "encryptionkey@0"
I20260619 14:46:42.259693 3301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781880383132139-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781880383132139-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781880383132139-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:46:42.277518 3301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:46:42.281461 3301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:46:42.281718 3301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:46:42.289597 3301 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42201
I20260619 14:46:42.289933 3368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42201 every 8 connection(s)
I20260619 14:46:42.290724 3301 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:46:42.293648 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 3301
I20260619 14:46:42.293782 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781880383132139-392-0/minicluster-data/master-0/wal/instance
I20260619 14:46:42.294104 392 external_mini_cluster.cc:1428] Setting key a08fcaf16ab66e34104e88ed308e8745
I20260619 14:46:42.294286 3369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:46:42.302918 3369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 328fe308b68741828fc99ee88d906788: Bootstrap starting.
Jun 19 14:46:42 dist-test-slave-2sb1 krb5kdc[3288](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880402, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:46:42.306083 3369 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 328fe308b68741828fc99ee88d906788: Neither blocks nor log segments found. Creating new log.
I20260619 14:46:42.307065 3369 log.cc:824] T 00000000000000000000000000000000 P 328fe308b68741828fc99ee88d906788: Log is configured to *not* fsync() on all Append() calls
I20260619 14:46:42.310274 3369 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 328fe308b68741828fc99ee88d906788: No bootstrap required, opened a new log
I20260619 14:46:42.314041 3372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:42.295783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55604 (local address 127.0.98.62:42201)
0619 14:46:42.296297 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:42.296314 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:42.296350 (+ 36us) server_negotiation.cc:407] Connection header received
0619 14:46:42.297088 (+ 738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:42.297108 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:42.297512 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:42.297835 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:42.298922 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:42.300169 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:42.301513 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:42.302044 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:42.302183 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:42.305567 (+ 3384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:42.305573 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:42.305589 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:42.305637 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:42.308273 (+ 2636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:42.309147 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:42.309153 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:42.309159 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:42.309289 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:42.309797 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:42.309800 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:42.309802 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:42.310711 (+ 909us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:42.311392 (+ 681us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:42.311639 (+ 247us) server_negotiation.cc:299] Negotiation successful
0619 14:46:42.312490 (+ 851us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":176,"threads_started":1}
I20260619 14:46:42.316077 3369 raft_consensus.cc:348] T 00000000000000000000000000000000 P 328fe308b68741828fc99ee88d906788 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "328fe308b68741828fc99ee88d906788" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42201 } }
I20260619 14:46:42.316843 3369 raft_consensus.cc:374] T 00000000000000000000000000000000 P 328fe308b68741828fc99ee88d906788 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:46:42.317039 3369 raft_consensus.cc:729] T 00000000000000000000000000000000 P 328fe308b68741828fc99ee88d906788 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 328fe308b68741828fc99ee88d906788, State: Initialized, Role: FOLLOWER
I20260619 14:46:42.317615 3369 consensus_queue.cc:260] T 00000000000000000000000000000000 P 328fe308b68741828fc99ee88d906788 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "328fe308b68741828fc99ee88d906788" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42201 } }
I20260619 14:46:42.317821 3369 raft_consensus.cc:388] T 00000000000000000000000000000000 P 328fe308b68741828fc99ee88d906788 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:46:42.317914 3369 raft_consensus.cc:482] T 00000000000000000000000000000000 P 328fe308b68741828fc99ee88d906788 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:46:42.318065 3369 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 328fe308b68741828fc99ee88d906788 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:46:42.319118 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:46:42.319391 3369 raft_consensus.cc:504] T 00000000000000000000000000000000 P 328fe308b68741828fc99ee88d906788 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "328fe308b68741828fc99ee88d906788" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42201 } }
I20260619 14:46:42.319836 3369 leader_election.cc:302] T 00000000000000000000000000000000 P 328fe308b68741828fc99ee88d906788 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 328fe308b68741828fc99ee88d906788; no voters:
I20260619 14:46:42.320253 3369 leader_election.cc:288] T 00000000000000000000000000000000 P 328fe308b68741828fc99ee88d906788 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:46:42.320546 3374 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 328fe308b68741828fc99ee88d906788 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:46:42.320886 3374 raft_consensus.cc:686] T 00000000000000000000000000000000 P 328fe308b68741828fc99ee88d906788 [term 1 LEADER]: Becoming Leader. State: Replica: 328fe308b68741828fc99ee88d906788, State: Running, Role: LEADER
I20260619 14:46:42.321283 3374 consensus_queue.cc:237] T 00000000000000000000000000000000 P 328fe308b68741828fc99ee88d906788 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "328fe308b68741828fc99ee88d906788" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42201 } }
I20260619 14:46:42.321776 3369 sys_catalog.cc:564] T 00000000000000000000000000000000 P 328fe308b68741828fc99ee88d906788 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:46:42.323401 3376 sys_catalog.cc:455] T 00000000000000000000000000000000 P 328fe308b68741828fc99ee88d906788 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 328fe308b68741828fc99ee88d906788. Latest consensus state: current_term: 1 leader_uuid: "328fe308b68741828fc99ee88d906788" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "328fe308b68741828fc99ee88d906788" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42201 } } }
I20260619 14:46:42.323581 3376 sys_catalog.cc:458] T 00000000000000000000000000000000 P 328fe308b68741828fc99ee88d906788 [sys.catalog]: This master's current role is: LEADER
I20260619 14:46:42.323812 3375 sys_catalog.cc:455] T 00000000000000000000000000000000 P 328fe308b68741828fc99ee88d906788 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "328fe308b68741828fc99ee88d906788" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "328fe308b68741828fc99ee88d906788" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42201 } } }
I20260619 14:46:42.323942 3375 sys_catalog.cc:458] T 00000000000000000000000000000000 P 328fe308b68741828fc99ee88d906788 [sys.catalog]: This master's current role is: LEADER
I20260619 14:46:42.324307 3384 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:46:42.327735 3384 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:46:42.334772 3384 catalog_manager.cc:1347] Generated new cluster ID: 9727584dd6c340d58edd59a571150ab6
I20260619 14:46:42.334875 3384 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:46:42.346619 3372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:42.341304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55614 (local address 127.0.98.62:42201)
0619 14:46:42.341445 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:42.341450 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:42.341727 (+ 277us) server_negotiation.cc:407] Connection header received
0619 14:46:42.341904 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:42.341908 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:42.341962 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:42.342056 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:42.342651 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:42.343256 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:42.344153 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:42.344416 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:42.344541 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:42.345592 (+ 1051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:42.345602 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:42.345606 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:46:42.346109 (+ 503us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:42.346144 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:46:42.346162 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:42.346283 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:42.346375 (+ 92us) server_negotiation.cc:299] Negotiation successful
0619 14:46:42.346449 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 14:46:42.377473 3384 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:46:42.378688 3384 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:46:42.385565 3384 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 328fe308b68741828fc99ee88d906788: Generated new TSK 0
I20260619 14:46:42.386372 3384 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:46:42.400775 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 3301
2026-06-19T14:46:42Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1627 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880383132139-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:46:42 dist-test-slave-2sb1 krb5kdc[3402](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:46:42 dist-test-slave-2sb1 krb5kdc[3402](info): set up 2 sockets
Jun 19 14:46:42 dist-test-slave-2sb1 krb5kdc[3402](info): commencing operation
krb5kdc: starting...
W20260619 14:46:43.564882 392 mini_kdc.cc:121] Time spent starting KDC: real 1.144s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880383132139-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880383132139-392-0/krb5kdc/test-user.keytab.
Jun 19 14:46:43 dist-test-slave-2sb1 krb5kdc[3402](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880403, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:46:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:46:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 14:46:43.713181 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880383132139-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880383132139-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880383132139-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39285
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39285
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:46:43.835417 3415 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:46:43.835732 3415 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:46:43.835775 3415 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:46:43.840441 3415 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:46:43.840533 3415 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:46:43.840553 3415 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:46:43.840571 3415 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:46:43.840586 3415 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:46:43.846493 3415 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880383132139-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880383132139-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39285
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39285
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3415
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880383132139-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:46:43.847662 3415 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:46:43.848785 3415 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:46:43.858806 3420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:43.858878 3421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:43.858817 3423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:43.859596 3415 server_base.cc:1034] running on GCE node
I20260619 14:46:43.860314 3415 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:46:43.861866 3415 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:46:43.863070 3415 hybrid_clock.cc:648] HybridClock initialized: now 1781880403863035 us; error 55 us; skew 500 ppm
Jun 19 14:46:43 dist-test-slave-2sb1 krb5kdc[3402](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880403, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:46:43.870901 3415 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 14:46:43.872181 3415 webserver.cc:460] Webserver started at http://127.0.98.62:37259/ using document root <none> and password file <none>
I20260619 14:46:43.873071 3415 fs_manager.cc:359] Metadata directory not provided
I20260619 14:46:43.873193 3415 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:46:43.873492 3415 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:46:43.876085 3415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880383132139-392-0/minicluster-data/master-0/data/instance:
uuid: "298f203e732546ecba0c315990f9aaf9"
format_stamp: "Formatted at 2026-06-19 14:46:43 on dist-test-slave-2sb1"
server_key: "8ff1156d9ad064ecf93d226de9043129"
server_key_iv: "9f0c1c0924b0be5b9a55341c9f1b1727"
server_key_version: "encryptionkey@0"
I20260619 14:46:43.876953 3415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880383132139-392-0/minicluster-data/master-0/wal/instance:
uuid: "298f203e732546ecba0c315990f9aaf9"
format_stamp: "Formatted at 2026-06-19 14:46:43 on dist-test-slave-2sb1"
server_key: "8ff1156d9ad064ecf93d226de9043129"
server_key_iv: "9f0c1c0924b0be5b9a55341c9f1b1727"
server_key_version: "encryptionkey@0"
I20260619 14:46:43.881392 3415 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 14:46:43.884407 3430 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:43.886091 3415 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:46:43.886284 3415 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880383132139-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880383132139-392-0/minicluster-data/master-0/wal
uuid: "298f203e732546ecba0c315990f9aaf9"
format_stamp: "Formatted at 2026-06-19 14:46:43 on dist-test-slave-2sb1"
server_key: "8ff1156d9ad064ecf93d226de9043129"
server_key_iv: "9f0c1c0924b0be5b9a55341c9f1b1727"
server_key_version: "encryptionkey@0"
I20260619 14:46:43.886466 3415 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880383132139-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880383132139-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880383132139-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:46:43.918118 3415 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:46:43.922081 3415 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:46:43.922315 3415 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:46:43.931020 3415 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39285
I20260619 14:46:43.931337 3482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39285 every 8 connection(s)
I20260619 14:46:43.932174 3415 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:46:43.935379 3483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:46:43.941464 3483 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 298f203e732546ecba0c315990f9aaf9: Bootstrap starting.
I20260619 14:46:43.941836 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 3415
I20260619 14:46:43.941990 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880383132139-392-0/minicluster-data/master-0/wal/instance
I20260619 14:46:43.942304 392 external_mini_cluster.cc:1428] Setting key a5db3f47b0fa4ec6d3170847c32e1b03
I20260619 14:46:43.944418 3483 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 298f203e732546ecba0c315990f9aaf9: Neither blocks nor log segments found. Creating new log.
I20260619 14:46:43.945257 3483 log.cc:824] T 00000000000000000000000000000000 P 298f203e732546ecba0c315990f9aaf9: Log is configured to *not* fsync() on all Append() calls
I20260619 14:46:43.947320 3483 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 298f203e732546ecba0c315990f9aaf9: No bootstrap required, opened a new log
I20260619 14:46:43.950724 3483 raft_consensus.cc:348] T 00000000000000000000000000000000 P 298f203e732546ecba0c315990f9aaf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "298f203e732546ecba0c315990f9aaf9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39285 } }
I20260619 14:46:43.951047 3483 raft_consensus.cc:374] T 00000000000000000000000000000000 P 298f203e732546ecba0c315990f9aaf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:46:43.951105 3483 raft_consensus.cc:729] T 00000000000000000000000000000000 P 298f203e732546ecba0c315990f9aaf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 298f203e732546ecba0c315990f9aaf9, State: Initialized, Role: FOLLOWER
I20260619 14:46:43.951608 3483 consensus_queue.cc:260] T 00000000000000000000000000000000 P 298f203e732546ecba0c315990f9aaf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "298f203e732546ecba0c315990f9aaf9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39285 } }
I20260619 14:46:43.951745 3483 raft_consensus.cc:388] T 00000000000000000000000000000000 P 298f203e732546ecba0c315990f9aaf9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:46:43.951850 3483 raft_consensus.cc:482] T 00000000000000000000000000000000 P 298f203e732546ecba0c315990f9aaf9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:46:43.951990 3483 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 298f203e732546ecba0c315990f9aaf9 [term 0 FOLLOWER]: Advancing to term 1
Jun 19 14:46:43 dist-test-slave-2sb1 krb5kdc[3402](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880403, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:46:43.953233 3483 raft_consensus.cc:504] T 00000000000000000000000000000000 P 298f203e732546ecba0c315990f9aaf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "298f203e732546ecba0c315990f9aaf9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39285 } }
I20260619 14:46:43.953699 3483 leader_election.cc:302] T 00000000000000000000000000000000 P 298f203e732546ecba0c315990f9aaf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 298f203e732546ecba0c315990f9aaf9; no voters:
I20260619 14:46:43.954144 3483 leader_election.cc:288] T 00000000000000000000000000000000 P 298f203e732546ecba0c315990f9aaf9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:46:43.954196 3488 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 298f203e732546ecba0c315990f9aaf9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:46:43.954407 3488 raft_consensus.cc:686] T 00000000000000000000000000000000 P 298f203e732546ecba0c315990f9aaf9 [term 1 LEADER]: Becoming Leader. State: Replica: 298f203e732546ecba0c315990f9aaf9, State: Running, Role: LEADER
I20260619 14:46:43.955130 3488 consensus_queue.cc:237] T 00000000000000000000000000000000 P 298f203e732546ecba0c315990f9aaf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "298f203e732546ecba0c315990f9aaf9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39285 } }
I20260619 14:46:43.955606 3483 sys_catalog.cc:564] T 00000000000000000000000000000000 P 298f203e732546ecba0c315990f9aaf9 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:46:43.957623 3489 sys_catalog.cc:455] T 00000000000000000000000000000000 P 298f203e732546ecba0c315990f9aaf9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 298f203e732546ecba0c315990f9aaf9. Latest consensus state: current_term: 1 leader_uuid: "298f203e732546ecba0c315990f9aaf9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "298f203e732546ecba0c315990f9aaf9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39285 } } }
I20260619 14:46:43.957674 3490 sys_catalog.cc:455] T 00000000000000000000000000000000 P 298f203e732546ecba0c315990f9aaf9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "298f203e732546ecba0c315990f9aaf9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "298f203e732546ecba0c315990f9aaf9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39285 } } }
I20260619 14:46:43.957820 3489 sys_catalog.cc:458] T 00000000000000000000000000000000 P 298f203e732546ecba0c315990f9aaf9 [sys.catalog]: This master's current role is: LEADER
I20260619 14:46:43.957824 3490 sys_catalog.cc:458] T 00000000000000000000000000000000 P 298f203e732546ecba0c315990f9aaf9 [sys.catalog]: This master's current role is: LEADER
W20260619 14:46:43.961664 3503 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 298f203e732546ecba0c315990f9aaf9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260619 14:46:43.961664 3486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:43.943812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56750 (local address 127.0.98.62:39285)
0619 14:46:43.944292 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:43.944309 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:43.944350 (+ 41us) server_negotiation.cc:407] Connection header received
0619 14:46:43.945213 (+ 863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:43.945235 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:43.945658 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:43.946063 (+ 405us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:43.946994 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:43.948144 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:43.949193 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:43.950443 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:43.950631 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:43.954130 (+ 3499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:43.954138 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:43.954155 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:43.954192 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:43.956843 (+ 2651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:43.957512 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:43.957519 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:43.957527 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:43.957712 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:43.958118 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:43.958122 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:43.958124 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:43.958948 (+ 824us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:43.960019 (+ 1071us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:43.960186 (+ 167us) server_negotiation.cc:299] Negotiation successful
0619 14:46:43.960539 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":155,"threads_started":1}
W20260619 14:46:43.961804 3503 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:46:43.962172 3502 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:46:43.963162 3502 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:46:43.970193 3502 catalog_manager.cc:1347] Generated new cluster ID: e706da220e71472ca3c7991e44360c97
I20260619 14:46:43.970293 3502 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:46:43.995908 3502 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:46:43.996994 3502 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:46:44.009600 3502 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 298f203e732546ecba0c315990f9aaf9: Generated new TSK 0
I20260619 14:46:44.010726 3502 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880383132139-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880383132139-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 14:46:44.061362 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880383132139-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880383132139-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880383132139-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880383132139-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39285
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880383132139-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:46:44.192927 3509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:46:44.193287 3509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:46:44.193364 3509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:46:44.197682 3509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:46:44.197813 3509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:46:44.197917 3509 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 14:46:44.203888 3509 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880383132139-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880383132139-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880383132139-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880383132139-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39285
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3509
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880383132139-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:46:44.205404 3509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:46:44.206626 3509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:46:44.217947 3514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:44.217926 3517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:44.217985 3509 server_base.cc:1034] running on GCE node
W20260619 14:46:44.217926 3515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:44.218531 3509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:46:44.219250 3509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:46:44.220464 3509 hybrid_clock.cc:648] HybridClock initialized: now 1781880404220435 us; error 38 us; skew 500 ppm
Jun 19 14:46:44 dist-test-slave-2sb1 krb5kdc[3402](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880404, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:46:44.224119 3509 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 14:46:44.225206 3509 webserver.cc:460] Webserver started at http://127.0.98.1:41269/ using document root <none> and password file <none>
I20260619 14:46:44.225934 3509 fs_manager.cc:359] Metadata directory not provided
I20260619 14:46:44.226042 3509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:46:44.226294 3509 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:46:44.228509 3509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880383132139-392-0/minicluster-data/ts-0/data/instance:
uuid: "f1cee621da1d46019f2d9a24b873edf4"
format_stamp: "Formatted at 2026-06-19 14:46:44 on dist-test-slave-2sb1"
server_key: "5abcdc9fe0b1d6a4e1ab76bc59350338"
server_key_iv: "39270d312e8317d9ab39570fe66417bc"
server_key_version: "encryptionkey@0"
I20260619 14:46:44.229211 3509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880383132139-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f1cee621da1d46019f2d9a24b873edf4"
format_stamp: "Formatted at 2026-06-19 14:46:44 on dist-test-slave-2sb1"
server_key: "5abcdc9fe0b1d6a4e1ab76bc59350338"
server_key_iv: "39270d312e8317d9ab39570fe66417bc"
server_key_version: "encryptionkey@0"
I20260619 14:46:44.235152 3509 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260619 14:46:44.238358 3524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:44.239804 3509 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 14:46:44.239977 3509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880383132139-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880383132139-392-0/minicluster-data/ts-0/wal
uuid: "f1cee621da1d46019f2d9a24b873edf4"
format_stamp: "Formatted at 2026-06-19 14:46:44 on dist-test-slave-2sb1"
server_key: "5abcdc9fe0b1d6a4e1ab76bc59350338"
server_key_iv: "39270d312e8317d9ab39570fe66417bc"
server_key_version: "encryptionkey@0"
I20260619 14:46:44.240125 3509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880383132139-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880383132139-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880383132139-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:46:44.301046 3509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:46:44.305493 3509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:46:44.305749 3509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:46:44.306427 3509 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:46:44.307402 3509 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:46:44.307485 3509 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:44.307561 3509 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:46:44.307605 3509 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:44.321008 3509 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43379
I20260619 14:46:44.321017 3637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43379 every 8 connection(s)
I20260619 14:46:44.322088 3509 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880383132139-392-0/minicluster-data/ts-0/data/info.pb
I20260619 14:46:44.331558 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 3509
I20260619 14:46:44.331676 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880383132139-392-0/minicluster-data/ts-0/wal/instance
I20260619 14:46:44.331992 392 external_mini_cluster.cc:1428] Setting key 7096f6b5ca9bfc8ecb815c96731f2912
Jun 19 14:46:44 dist-test-slave-2sb1 krb5kdc[3402](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880404, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:46:44.339854 3486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:44.324404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53359 (local address 127.0.98.62:39285)
0619 14:46:44.324586 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:44.324590 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:44.325774 (+ 1184us) server_negotiation.cc:407] Connection header received
0619 14:46:44.327452 (+ 1678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:44.327456 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:44.327524 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:44.327646 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:44.329127 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:44.329724 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:44.331071 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:44.331341 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:44.331484 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:44.335023 (+ 3539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:44.335031 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:44.335038 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:44.335079 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:44.337123 (+ 2044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:44.337900 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:44.337909 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:44.337911 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:44.337995 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:44.338520 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:44.338524 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:44.338525 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:44.338735 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:44.338860 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:44.339503 (+ 643us) server_negotiation.cc:299] Negotiation successful
0619 14:46:44.339666 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260619 14:46:44.341058 3640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:44.324858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39285 (local address 127.0.98.1:53359)
0619 14:46:44.325569 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:44.325620 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:44.327149 (+ 1529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:44.327854 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:44.327862 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:44.328278 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:46:44.328921 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:44.328938 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:44.329892 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:44.329896 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:44.330908 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:44.330919 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:44.331548 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:44.331553 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:44.331818 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:44.332593 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:46:44.332622 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:46:44.334783 (+ 2161us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:46:44.337389 (+ 2606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:44.337399 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:44.337419 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:44.337758 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:44.338107 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:44.338110 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:44.338112 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:44.338282 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:44.338861 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:46:44.338870 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:46:44.339230 (+ 360us) client_negotiation.cc:769] Sending connection context
0619 14:46:44.339449 (+ 219us) client_negotiation.cc:240] Negotiation successful
0619 14:46:44.339815 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":157,"threads_started":1}
I20260619 14:46:44.342672 3638 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39285
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 14:46:44.342962 3638 heartbeater.cc:464] Registering TS with master...
I20260619 14:46:44.343497 3638 heartbeater.cc:511] Master 127.0.98.62:39285 requested a full tablet report, sending...
I20260619 14:46:44.345530 3447 ts_manager.cc:194] Registered new tserver with Master: f1cee621da1d46019f2d9a24b873edf4 (127.0.98.1:43379)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 14:46:44.346961 3447 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:53359
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880383132139-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880383132139-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 14:46:44.370904 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880383132139-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880383132139-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880383132139-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880383132139-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39285
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880383132139-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:46:44.497642 3643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:46:44.497948 3643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:46:44.497995 3643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:46:44.502377 3643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:46:44.502465 3643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:46:44.502528 3643 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 14:46:44.509207 3643 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880383132139-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880383132139-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880383132139-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880383132139-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39285
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3643
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880383132139-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:46:44.510473 3643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:46:44.511675 3643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:46:44.524114 3649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:44.524132 3648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:44.524132 3651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:44.524196 3643 server_base.cc:1034] running on GCE node
I20260619 14:46:44.525116 3643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:46:44.525830 3643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:46:44.527109 3643 hybrid_clock.cc:648] HybridClock initialized: now 1781880404527069 us; error 56 us; skew 500 ppm
Jun 19 14:46:44 dist-test-slave-2sb1 krb5kdc[3402](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880404, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:46:44.530658 3643 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 14:46:44.531746 3643 webserver.cc:460] Webserver started at http://127.0.98.2:35287/ using document root <none> and password file <none>
I20260619 14:46:44.532373 3643 fs_manager.cc:359] Metadata directory not provided
I20260619 14:46:44.532450 3643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:46:44.532631 3643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:46:44.535653 3643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880383132139-392-0/minicluster-data/ts-1/data/instance:
uuid: "f4c172b9c7cb475498dd7b48081690ea"
format_stamp: "Formatted at 2026-06-19 14:46:44 on dist-test-slave-2sb1"
server_key: "20c80db623f1e23c5664aab49c906f91"
server_key_iv: "16fbc025433e2e625aae62f491a95efe"
server_key_version: "encryptionkey@0"
I20260619 14:46:44.536439 3643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880383132139-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f4c172b9c7cb475498dd7b48081690ea"
format_stamp: "Formatted at 2026-06-19 14:46:44 on dist-test-slave-2sb1"
server_key: "20c80db623f1e23c5664aab49c906f91"
server_key_iv: "16fbc025433e2e625aae62f491a95efe"
server_key_version: "encryptionkey@0"
I20260619 14:46:44.541235 3643 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 14:46:44.544198 3658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:44.546100 3643 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:46:44.546245 3643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880383132139-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880383132139-392-0/minicluster-data/ts-1/wal
uuid: "f4c172b9c7cb475498dd7b48081690ea"
format_stamp: "Formatted at 2026-06-19 14:46:44 on dist-test-slave-2sb1"
server_key: "20c80db623f1e23c5664aab49c906f91"
server_key_iv: "16fbc025433e2e625aae62f491a95efe"
server_key_version: "encryptionkey@0"
I20260619 14:46:44.546355 3643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880383132139-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880383132139-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880383132139-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:46:44.558709 3643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:46:44.562911 3643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:46:44.563105 3643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:46:44.563722 3643 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:46:44.564754 3643 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:46:44.564828 3643 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:44.564883 3643 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:46:44.564901 3643 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:44.575991 3643 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40521
I20260619 14:46:44.576015 3771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40521 every 8 connection(s)
I20260619 14:46:44.577142 3643 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880383132139-392-0/minicluster-data/ts-1/data/info.pb
I20260619 14:46:44.578684 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 3643
I20260619 14:46:44.578925 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880383132139-392-0/minicluster-data/ts-1/wal/instance
I20260619 14:46:44.579250 392 external_mini_cluster.cc:1428] Setting key 0ae2279c09dbc8167c4e809eb6ba45bb
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Jun 19 14:46:44 dist-test-slave-2sb1 krb5kdc[3402](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880404, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:46:44.603986 3486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:44.579707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39427 (local address 127.0.98.62:39285)
0619 14:46:44.580033 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:44.580041 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:44.582657 (+ 2616us) server_negotiation.cc:407] Connection header received
0619 14:46:44.585254 (+ 2597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:44.585260 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:44.585522 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:44.585674 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:44.588024 (+ 2350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:44.588887 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:44.591279 (+ 2392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:44.591590 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:44.591698 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:44.599568 (+ 7870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:44.599572 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:44.599574 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:44.599605 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:44.601523 (+ 1918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:44.602264 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:44.602267 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:44.602268 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:44.602354 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:44.602768 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:44.602772 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:44.602774 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:44.602983 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:44.603097 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:44.603670 (+ 573us) server_negotiation.cc:299] Negotiation successful
0619 14:46:44.603814 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225}
I20260619 14:46:44.604977 3774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:44.580993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39285 (local address 127.0.98.2:39427)
0619 14:46:44.582082 (+ 1089us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:44.582152 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:44.584746 (+ 2594us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:44.586183 (+ 1437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:44.586201 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:44.586893 (+ 692us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:46:44.587742 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:44.587773 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:44.589078 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:44.589085 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:44.591068 (+ 1983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:44.591091 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:44.591810 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:44.591815 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:44.592100 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:44.593072 (+ 972us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:46:44.593104 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:46:44.599287 (+ 6183us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:46:44.601693 (+ 2406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:44.601701 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:44.601741 (+ 40us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:44.602103 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:44.602469 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:44.602472 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:44.602474 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:44.602652 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:44.603099 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:46:44.603108 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:46:44.603398 (+ 290us) client_negotiation.cc:769] Sending connection context
0619 14:46:44.603630 (+ 232us) client_negotiation.cc:240] Negotiation successful
0619 14:46:44.604014 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":220,"threads_started":1}
I20260619 14:46:44.606545 3772 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39285
I20260619 14:46:44.606874 3772 heartbeater.cc:464] Registering TS with master...
I20260619 14:46:44.607568 3772 heartbeater.cc:511] Master 127.0.98.62:39285 requested a full tablet report, sending...
I20260619 14:46:44.609023 3447 ts_manager.cc:194] Registered new tserver with Master: f4c172b9c7cb475498dd7b48081690ea (127.0.98.2:40521)
I20260619 14:46:44.609863 3447 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:39427
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880383132139-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880383132139-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 14:46:44.621802 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880383132139-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880383132139-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880383132139-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880383132139-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39285
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880383132139-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:46:44.741779 3777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:46:44.742136 3777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:46:44.742182 3777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:46:44.746470 3777 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:46:44.746548 3777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:46:44.746610 3777 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 14:46:44.752374 3777 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880383132139-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880383132139-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880383132139-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880383132139-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39285
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3777
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880383132139-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:46:44.753743 3777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:46:44.754810 3777 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:46:44.766368 3782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:44.767980 3785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:44.768152 3777 server_base.cc:1034] running on GCE node
W20260619 14:46:44.768327 3783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:44.768944 3777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:46:44.769752 3777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:46:44.770982 3777 hybrid_clock.cc:648] HybridClock initialized: now 1781880404770943 us; error 53 us; skew 500 ppm
Jun 19 14:46:44 dist-test-slave-2sb1 krb5kdc[3402](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880404, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:46:44.774492 3777 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 14:46:44.775657 3777 webserver.cc:460] Webserver started at http://127.0.98.3:44669/ using document root <none> and password file <none>
I20260619 14:46:44.776366 3777 fs_manager.cc:359] Metadata directory not provided
I20260619 14:46:44.776472 3777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:46:44.776793 3777 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:46:44.779103 3777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880383132139-392-0/minicluster-data/ts-2/data/instance:
uuid: "3978c8123b804d0bb09a93329dfce2d5"
format_stamp: "Formatted at 2026-06-19 14:46:44 on dist-test-slave-2sb1"
server_key: "aeac35e16fa87d405c370577e2a937d6"
server_key_iv: "b543166ef4bb7354fdc3c909ef89de20"
server_key_version: "encryptionkey@0"
I20260619 14:46:44.779714 3777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880383132139-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3978c8123b804d0bb09a93329dfce2d5"
format_stamp: "Formatted at 2026-06-19 14:46:44 on dist-test-slave-2sb1"
server_key: "aeac35e16fa87d405c370577e2a937d6"
server_key_iv: "b543166ef4bb7354fdc3c909ef89de20"
server_key_version: "encryptionkey@0"
I20260619 14:46:44.784318 3777 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 14:46:44.787407 3792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:44.789077 3777 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:46:44.789245 3777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880383132139-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880383132139-392-0/minicluster-data/ts-2/wal
uuid: "3978c8123b804d0bb09a93329dfce2d5"
format_stamp: "Formatted at 2026-06-19 14:46:44 on dist-test-slave-2sb1"
server_key: "aeac35e16fa87d405c370577e2a937d6"
server_key_iv: "b543166ef4bb7354fdc3c909ef89de20"
server_key_version: "encryptionkey@0"
I20260619 14:46:44.789533 3777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880383132139-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880383132139-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880383132139-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:46:44.807183 3777 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:46:44.810694 3777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:46:44.810875 3777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:46:44.811599 3777 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:46:44.812450 3777 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:46:44.812498 3777 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:44.812541 3777 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:46:44.812556 3777 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:44.823858 3777 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40215
I20260619 14:46:44.823875 3905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40215 every 8 connection(s)
I20260619 14:46:44.825045 3777 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880383132139-392-0/minicluster-data/ts-2/data/info.pb
I20260619 14:46:44.829675 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 3777
I20260619 14:46:44.829859 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781880383132139-392-0/minicluster-data/ts-2/wal/instance
I20260619 14:46:44.830169 392 external_mini_cluster.cc:1428] Setting key 84861fcb4582576a761d2f5dc8831dfc
Jun 19 14:46:44 dist-test-slave-2sb1 krb5kdc[3402](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880404, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:46:44.846501 3486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:44.827319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37865 (local address 127.0.98.62:39285)
0619 14:46:44.827487 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:44.827491 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:44.828997 (+ 1506us) server_negotiation.cc:407] Connection header received
0619 14:46:44.830425 (+ 1428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:44.830429 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:44.830488 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:44.830601 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:44.831988 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:44.832761 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:44.834139 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:44.834441 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:44.834551 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:44.841374 (+ 6823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:44.841377 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:44.841379 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:44.841409 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:44.843360 (+ 1951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:44.844168 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:44.844172 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:44.844175 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:44.844445 (+ 270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:44.844923 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:44.844927 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:44.844929 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:44.845187 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:44.845310 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:44.846112 (+ 802us) server_negotiation.cc:299] Negotiation successful
0619 14:46:44.846374 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 14:46:44.847385 3908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:44.827944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39285 (local address 127.0.98.3:37865)
0619 14:46:44.828762 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:44.828811 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:44.830182 (+ 1371us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:44.830786 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:44.830794 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:44.831289 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:46:44.831825 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:44.831839 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:44.832919 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:44.832923 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:44.833994 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:44.834004 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:44.834627 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:44.834631 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:44.834808 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:44.838719 (+ 3911us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:46:44.838746 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:46:44.841126 (+ 2380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:46:44.843543 (+ 2417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:44.843551 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:44.843572 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:44.844018 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:44.844557 (+ 539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:44.844560 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:44.844562 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:44.844810 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:44.845406 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:46:44.845413 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:46:44.845797 (+ 384us) client_negotiation.cc:769] Sending connection context
0619 14:46:44.846003 (+ 206us) client_negotiation.cc:240] Negotiation successful
0619 14:46:44.846470 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":226,"threads_started":1}
I20260619 14:46:44.849103 3906 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39285
I20260619 14:46:44.849407 3906 heartbeater.cc:464] Registering TS with master...
I20260619 14:46:44.849902 3906 heartbeater.cc:511] Master 127.0.98.62:39285 requested a full tablet report, sending...
I20260619 14:46:44.851037 3447 ts_manager.cc:194] Registered new tserver with Master: 3978c8123b804d0bb09a93329dfce2d5 (127.0.98.3:40215)
I20260619 14:46:44.851871 3447 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:37865
I20260619 14:46:44.857962 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 14:46:44 dist-test-slave-2sb1 krb5kdc[3402](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880404, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 19 14:46:44 dist-test-slave-2sb1 krb5kdc[3402](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880404, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:46:44.889667 3486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:44.879204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56764 (local address 127.0.98.62:39285)
0619 14:46:44.879580 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:44.879588 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:44.879701 (+ 113us) server_negotiation.cc:407] Connection header received
0619 14:46:44.879934 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:44.879937 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:44.879985 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:44.880087 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:44.881013 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:44.881687 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:44.882685 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:44.882939 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:44.883061 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:44.886169 (+ 3108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:44.886173 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:44.886175 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:44.886299 (+ 124us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:44.887800 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:44.888278 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:44.888282 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:44.888284 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:44.888373 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:44.888704 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:44.888707 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:44.888710 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:44.888982 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:44.889128 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:44.889372 (+ 244us) server_negotiation.cc:299] Negotiation successful
0619 14:46:44.889494 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263}
W20260619 14:46:44.890106 3447 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:56764
I20260619 14:46:44.891501 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 3509
I20260619 14:46:44.898083 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 3643
I20260619 14:46:44.904275 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 3777
I20260619 14:46:44.909798 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 3415
2026-06-19T14:46:44Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2511 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880383132139-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:46:44 dist-test-slave-2sb1 krb5kdc[3918](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:46:44 dist-test-slave-2sb1 krb5kdc[3918](info): set up 2 sockets
Jun 19 14:46:44 dist-test-slave-2sb1 krb5kdc[3918](info): commencing operation
krb5kdc: starting...
W20260619 14:46:46.033850 392 mini_kdc.cc:121] Time spent starting KDC: real 1.101s user 0.006s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880383132139-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880383132139-392-0/krb5kdc/test-user.keytab.
Jun 19 14:46:46 dist-test-slave-2sb1 krb5kdc[3918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880406, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:46:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:46:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 14:46:46.182756 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880383132139-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880383132139-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880383132139-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42045
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42045
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:46:46.306196 3931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:46:46.306565 3931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:46:46.306646 3931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:46:46.312346 3931 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:46:46.312462 3931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:46:46.312500 3931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:46:46.312538 3931 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:46:46.312572 3931 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:46:46.318605 3931 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880383132139-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880383132139-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42045
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42045
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3931
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880383132139-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:46:46.319828 3931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:46:46.321044 3931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:46:46.330673 3936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:46.330708 3937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:46.330910 3939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:46.331498 3931 server_base.cc:1034] running on GCE node
I20260619 14:46:46.332232 3931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:46:46.333645 3931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:46:46.334951 3931 hybrid_clock.cc:648] HybridClock initialized: now 1781880406334904 us; error 58 us; skew 500 ppm
Jun 19 14:46:46 dist-test-slave-2sb1 krb5kdc[3918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880406, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:46:46.338944 3931 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 14:46:46.340112 3931 webserver.cc:460] Webserver started at http://127.0.98.62:46113/ using document root <none> and password file <none>
I20260619 14:46:46.340827 3931 fs_manager.cc:359] Metadata directory not provided
I20260619 14:46:46.340930 3931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:46:46.341178 3931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:46:46.343365 3931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880383132139-392-0/minicluster-data/master-0/data/instance:
uuid: "a820f825d4674c34b05a395d9783a5a3"
format_stamp: "Formatted at 2026-06-19 14:46:46 on dist-test-slave-2sb1"
server_key: "a68b00227edb0bea411c7702e7ba2c05"
server_key_iv: "92742d32bb4ea8827319e28eb9cefc18"
server_key_version: "encryptionkey@0"
I20260619 14:46:46.343988 3931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880383132139-392-0/minicluster-data/master-0/wal/instance:
uuid: "a820f825d4674c34b05a395d9783a5a3"
format_stamp: "Formatted at 2026-06-19 14:46:46 on dist-test-slave-2sb1"
server_key: "a68b00227edb0bea411c7702e7ba2c05"
server_key_iv: "92742d32bb4ea8827319e28eb9cefc18"
server_key_version: "encryptionkey@0"
I20260619 14:46:46.348119 3931 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 14:46:46.351732 3946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:46.353575 3931 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260619 14:46:46.353761 3931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880383132139-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880383132139-392-0/minicluster-data/master-0/wal
uuid: "a820f825d4674c34b05a395d9783a5a3"
format_stamp: "Formatted at 2026-06-19 14:46:46 on dist-test-slave-2sb1"
server_key: "a68b00227edb0bea411c7702e7ba2c05"
server_key_iv: "92742d32bb4ea8827319e28eb9cefc18"
server_key_version: "encryptionkey@0"
I20260619 14:46:46.353901 3931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880383132139-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880383132139-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880383132139-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:46:46.384548 3931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:46:46.388700 3931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:46:46.388953 3931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:46:46.397388 3931 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42045
I20260619 14:46:46.397478 3998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42045 every 8 connection(s)
I20260619 14:46:46.398545 3931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:46:46.400609 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 3931
I20260619 14:46:46.400791 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880383132139-392-0/minicluster-data/master-0/wal/instance
I20260619 14:46:46.401137 392 external_mini_cluster.cc:1428] Setting key 8ca12a0854f121c06b365d28cd90062f
I20260619 14:46:46.401793 3999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:46:46.409017 3999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a820f825d4674c34b05a395d9783a5a3: Bootstrap starting.
Jun 19 14:46:46 dist-test-slave-2sb1 krb5kdc[3918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880406, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:46:46.412214 3999 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a820f825d4674c34b05a395d9783a5a3: Neither blocks nor log segments found. Creating new log.
I20260619 14:46:46.413203 3999 log.cc:824] T 00000000000000000000000000000000 P a820f825d4674c34b05a395d9783a5a3: Log is configured to *not* fsync() on all Append() calls
I20260619 14:46:46.415313 3999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a820f825d4674c34b05a395d9783a5a3: No bootstrap required, opened a new log
I20260619 14:46:46.417351 4002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:46.402581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59298 (local address 127.0.98.62:42045)
0619 14:46:46.403014 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:46.403032 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:46.403068 (+ 36us) server_negotiation.cc:407] Connection header received
0619 14:46:46.403732 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:46.403748 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:46.404078 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:46.404373 (+ 295us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:46.405520 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:46.406543 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:46.407732 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:46.408238 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:46.408364 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:46.411453 (+ 3089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:46.411459 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:46.411472 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:46.411505 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:46.413966 (+ 2461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:46.414520 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:46.414525 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:46.414530 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:46.414648 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:46.414966 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:46.414972 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:46.414978 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:46.415391 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:46.415617 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:46.415947 (+ 330us) server_negotiation.cc:299] Negotiation successful
0619 14:46:46.416302 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":150,"threads_started":1}
I20260619 14:46:46.418833 3999 raft_consensus.cc:348] T 00000000000000000000000000000000 P a820f825d4674c34b05a395d9783a5a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a820f825d4674c34b05a395d9783a5a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42045 } }
I20260619 14:46:46.419094 3999 raft_consensus.cc:374] T 00000000000000000000000000000000 P a820f825d4674c34b05a395d9783a5a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:46:46.419188 3999 raft_consensus.cc:729] T 00000000000000000000000000000000 P a820f825d4674c34b05a395d9783a5a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a820f825d4674c34b05a395d9783a5a3, State: Initialized, Role: FOLLOWER
I20260619 14:46:46.419668 3999 consensus_queue.cc:260] T 00000000000000000000000000000000 P a820f825d4674c34b05a395d9783a5a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a820f825d4674c34b05a395d9783a5a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42045 } }
I20260619 14:46:46.419826 3999 raft_consensus.cc:388] T 00000000000000000000000000000000 P a820f825d4674c34b05a395d9783a5a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:46:46.419916 3999 raft_consensus.cc:482] T 00000000000000000000000000000000 P a820f825d4674c34b05a395d9783a5a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:46:46.420019 3999 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a820f825d4674c34b05a395d9783a5a3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:46:46.421119 3999 raft_consensus.cc:504] T 00000000000000000000000000000000 P a820f825d4674c34b05a395d9783a5a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a820f825d4674c34b05a395d9783a5a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42045 } }
I20260619 14:46:46.421497 3999 leader_election.cc:302] T 00000000000000000000000000000000 P a820f825d4674c34b05a395d9783a5a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a820f825d4674c34b05a395d9783a5a3; no voters:
I20260619 14:46:46.421829 3999 leader_election.cc:288] T 00000000000000000000000000000000 P a820f825d4674c34b05a395d9783a5a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:46:46.422036 4004 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a820f825d4674c34b05a395d9783a5a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:46:46.422338 4004 raft_consensus.cc:686] T 00000000000000000000000000000000 P a820f825d4674c34b05a395d9783a5a3 [term 1 LEADER]: Becoming Leader. State: Replica: a820f825d4674c34b05a395d9783a5a3, State: Running, Role: LEADER
I20260619 14:46:46.422657 4004 consensus_queue.cc:237] T 00000000000000000000000000000000 P a820f825d4674c34b05a395d9783a5a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a820f825d4674c34b05a395d9783a5a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42045 } }
I20260619 14:46:46.423125 3999 sys_catalog.cc:564] T 00000000000000000000000000000000 P a820f825d4674c34b05a395d9783a5a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:46:46.424573 4006 sys_catalog.cc:455] T 00000000000000000000000000000000 P a820f825d4674c34b05a395d9783a5a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a820f825d4674c34b05a395d9783a5a3. Latest consensus state: current_term: 1 leader_uuid: "a820f825d4674c34b05a395d9783a5a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a820f825d4674c34b05a395d9783a5a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42045 } } }
I20260619 14:46:46.424743 4006 sys_catalog.cc:458] T 00000000000000000000000000000000 P a820f825d4674c34b05a395d9783a5a3 [sys.catalog]: This master's current role is: LEADER
I20260619 14:46:46.425303 4013 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:46:46.425469 4005 sys_catalog.cc:455] T 00000000000000000000000000000000 P a820f825d4674c34b05a395d9783a5a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a820f825d4674c34b05a395d9783a5a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a820f825d4674c34b05a395d9783a5a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42045 } } }
I20260619 14:46:46.425573 4005 sys_catalog.cc:458] T 00000000000000000000000000000000 P a820f825d4674c34b05a395d9783a5a3 [sys.catalog]: This master's current role is: LEADER
I20260619 14:46:46.428565 4013 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:46:46.435206 4013 catalog_manager.cc:1347] Generated new cluster ID: 7e61a29e90a548c39a90a281dbd31281
I20260619 14:46:46.435441 4013 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:46:46.451139 4013 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:46:46.452478 4013 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:46:46.458849 4013 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a820f825d4674c34b05a395d9783a5a3: Generated new TSK 0
I20260619 14:46:46.459860 4013 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880383132139-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880383132139-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 14:46:46.497494 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880383132139-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880383132139-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880383132139-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880383132139-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42045
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880383132139-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:46:46.621102 4025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:46:46.621433 4025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:46:46.621479 4025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:46:46.626251 4025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:46:46.626335 4025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:46:46.626675 4025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 14:46:46.633643 4025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880383132139-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880383132139-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880383132139-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880383132139-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42045
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4025
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880383132139-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:46:46.634897 4025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:46:46.636041 4025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:46:46.647099 4031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:46.647140 4033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:46.647099 4030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:46.647465 4025 server_base.cc:1034] running on GCE node
I20260619 14:46:46.647927 4025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:46:46.648885 4025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:46:46.650103 4025 hybrid_clock.cc:648] HybridClock initialized: now 1781880406650067 us; error 53 us; skew 500 ppm
Jun 19 14:46:46 dist-test-slave-2sb1 krb5kdc[3918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880406, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:46:46.653749 4025 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 14:46:46.654824 4025 webserver.cc:460] Webserver started at http://127.0.98.1:40659/ using document root <none> and password file <none>
I20260619 14:46:46.655444 4025 fs_manager.cc:359] Metadata directory not provided
I20260619 14:46:46.655542 4025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:46:46.655795 4025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:46:46.658289 4025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880383132139-392-0/minicluster-data/ts-0/data/instance:
uuid: "6f6a34b43dc846e585a18bad39f3c4e5"
format_stamp: "Formatted at 2026-06-19 14:46:46 on dist-test-slave-2sb1"
server_key: "4d7134d24ca3aac3faf8ab2b5880fb52"
server_key_iv: "dfe6098787473d0b827e9f955fa9e3bf"
server_key_version: "encryptionkey@0"
I20260619 14:46:46.658982 4025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880383132139-392-0/minicluster-data/ts-0/wal/instance:
uuid: "6f6a34b43dc846e585a18bad39f3c4e5"
format_stamp: "Formatted at 2026-06-19 14:46:46 on dist-test-slave-2sb1"
server_key: "4d7134d24ca3aac3faf8ab2b5880fb52"
server_key_iv: "dfe6098787473d0b827e9f955fa9e3bf"
server_key_version: "encryptionkey@0"
I20260619 14:46:46.663357 4025 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260619 14:46:46.666424 4040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:46.668254 4025 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 14:46:46.668546 4025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880383132139-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880383132139-392-0/minicluster-data/ts-0/wal
uuid: "6f6a34b43dc846e585a18bad39f3c4e5"
format_stamp: "Formatted at 2026-06-19 14:46:46 on dist-test-slave-2sb1"
server_key: "4d7134d24ca3aac3faf8ab2b5880fb52"
server_key_iv: "dfe6098787473d0b827e9f955fa9e3bf"
server_key_version: "encryptionkey@0"
I20260619 14:46:46.668804 4025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880383132139-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880383132139-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880383132139-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:46:46.694818 4025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:46:46.698738 4025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:46:46.698997 4025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:46:46.699633 4025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:46:46.700559 4025 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:46:46.700639 4025 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:46.700732 4025 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:46:46.700795 4025 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:46.711928 4025 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34741
I20260619 14:46:46.712010 4153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34741 every 8 connection(s)
I20260619 14:46:46.713033 4025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880383132139-392-0/minicluster-data/ts-0/data/info.pb
I20260619 14:46:46.715670 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 4025
I20260619 14:46:46.715849 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880383132139-392-0/minicluster-data/ts-0/wal/instance
I20260619 14:46:46.716150 392 external_mini_cluster.cc:1428] Setting key 675b1ef8668980e9d0d2810172aad178
Jun 19 14:46:46 dist-test-slave-2sb1 krb5kdc[3918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880406, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 14:46:46.732718 4002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:46.715460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52083 (local address 127.0.98.62:42045)
0619 14:46:46.715639 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:46.715644 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:46.716701 (+ 1057us) server_negotiation.cc:407] Connection header received
0619 14:46:46.718442 (+ 1741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:46.718446 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:46.718503 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:46.718599 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:46.720168 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:46.720848 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:46.722907 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:46.723199 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:46.723306 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:46.727346 (+ 4040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:46.727350 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:46.727352 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:46.727379 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:46.729472 (+ 2093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:46.730115 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:46.730140 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:46.730146 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:46.730393 (+ 247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:46.730923 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:46.730928 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:46.730930 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:46.731263 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:46.731395 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:46.732235 (+ 840us) server_negotiation.cc:299] Negotiation successful
0619 14:46:46.732425 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 14:46:46.734364 4156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:46.715880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42045 (local address 127.0.98.1:52083)
0619 14:46:46.716436 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:46.716499 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:46.718211 (+ 1712us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:46.718854 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:46.718862 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:46.719271 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:46:46.719922 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:46.719945 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:46.721002 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:46.721009 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:46.722609 (+ 1600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:46.722624 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:46.723399 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:46.723406 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:46.723702 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:46.724759 (+ 1057us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:46:46.724787 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:46:46.727113 (+ 2326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:46:46.729637 (+ 2524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:46.729644 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:46.729660 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:46.729980 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:46.730521 (+ 541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:46.730528 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:46.730533 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:46.730769 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:46.731516 (+ 747us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:46:46.731526 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:46:46.731905 (+ 379us) client_negotiation.cc:769] Sending connection context
0619 14:46:46.732168 (+ 263us) client_negotiation.cc:240] Negotiation successful
0619 14:46:46.732764 (+ 596us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":155,"threads_started":1}
I20260619 14:46:46.735913 4154 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42045
I20260619 14:46:46.736227 4154 heartbeater.cc:464] Registering TS with master...
I20260619 14:46:46.736963 4154 heartbeater.cc:511] Master 127.0.98.62:42045 requested a full tablet report, sending...
I20260619 14:46:46.738762 3963 ts_manager.cc:194] Registered new tserver with Master: 6f6a34b43dc846e585a18bad39f3c4e5 (127.0.98.1:34741)
I20260619 14:46:46.740114 3963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52083
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880383132139-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880383132139-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 14:46:46.754532 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880383132139-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880383132139-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880383132139-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880383132139-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42045
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880383132139-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:46:46.879052 4159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:46:46.879364 4159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:46:46.879413 4159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:46:46.884454 4159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:46:46.884564 4159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:46:46.884637 4159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 14:46:46.891098 4159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880383132139-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880383132139-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880383132139-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880383132139-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42045
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4159
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880383132139-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:46:46.892748 4159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:46:46.893915 4159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:46:46.905931 4165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:46.906203 4159 server_base.cc:1034] running on GCE node
W20260619 14:46:46.905964 4164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:46.906116 4167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:46.906908 4159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:46:46.907608 4159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:46:46.908754 4159 hybrid_clock.cc:648] HybridClock initialized: now 1781880406908740 us; error 53 us; skew 500 ppm
Jun 19 14:46:46 dist-test-slave-2sb1 krb5kdc[3918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880406, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:46:46.911957 4159 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 14:46:46.913012 4159 webserver.cc:460] Webserver started at http://127.0.98.2:34541/ using document root <none> and password file <none>
I20260619 14:46:46.913594 4159 fs_manager.cc:359] Metadata directory not provided
I20260619 14:46:46.913657 4159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:46:46.913834 4159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:46:46.915977 4159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880383132139-392-0/minicluster-data/ts-1/data/instance:
uuid: "12ee725b38344d839926962afefff5b6"
format_stamp: "Formatted at 2026-06-19 14:46:46 on dist-test-slave-2sb1"
server_key: "0362de5521a5c804301212f1e8bf5acf"
server_key_iv: "090bd8114da43348dc2518cce3c153e4"
server_key_version: "encryptionkey@0"
I20260619 14:46:46.916631 4159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880383132139-392-0/minicluster-data/ts-1/wal/instance:
uuid: "12ee725b38344d839926962afefff5b6"
format_stamp: "Formatted at 2026-06-19 14:46:46 on dist-test-slave-2sb1"
server_key: "0362de5521a5c804301212f1e8bf5acf"
server_key_iv: "090bd8114da43348dc2518cce3c153e4"
server_key_version: "encryptionkey@0"
I20260619 14:46:46.921082 4159 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 14:46:46.924733 4174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:46.926640 4159 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260619 14:46:46.926831 4159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880383132139-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880383132139-392-0/minicluster-data/ts-1/wal
uuid: "12ee725b38344d839926962afefff5b6"
format_stamp: "Formatted at 2026-06-19 14:46:46 on dist-test-slave-2sb1"
server_key: "0362de5521a5c804301212f1e8bf5acf"
server_key_iv: "090bd8114da43348dc2518cce3c153e4"
server_key_version: "encryptionkey@0"
I20260619 14:46:46.926995 4159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880383132139-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880383132139-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880383132139-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:46:46.969993 4159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:46:46.974926 4159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:46:46.975178 4159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:46:46.975878 4159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:46:46.976874 4159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:46:46.976938 4159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:46.977005 4159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:46:46.977039 4159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:46.987967 4159 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34279
I20260619 14:46:46.987998 4287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34279 every 8 connection(s)
I20260619 14:46:46.989176 4159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880383132139-392-0/minicluster-data/ts-1/data/info.pb
I20260619 14:46:46.993669 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 4159
I20260619 14:46:46.993815 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880383132139-392-0/minicluster-data/ts-1/wal/instance
I20260619 14:46:46.994127 392 external_mini_cluster.cc:1428] Setting key 2948f47f0b8fe22e1a3838dbc29570e5
Jun 19 14:46:46 dist-test-slave-2sb1 krb5kdc[3918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880406, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 14:46:47.006572 4002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:46.991199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45797 (local address 127.0.98.62:42045)
0619 14:46:46.991426 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:46.991430 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:46.992329 (+ 899us) server_negotiation.cc:407] Connection header received
0619 14:46:46.993909 (+ 1580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:46.993913 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:46.993973 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:46.994050 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:46.995620 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:46.996152 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:46.997703 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:46.997961 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:46.998070 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:47.001847 (+ 3777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:47.001855 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:47.001861 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:47.001906 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:47.003917 (+ 2011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:47.004728 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:47.004735 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:47.004739 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:47.004845 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:47.005341 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:47.005344 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:47.005345 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:47.005544 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:47.005676 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:47.006240 (+ 564us) server_negotiation.cc:299] Negotiation successful
0619 14:46:47.006396 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133}
I20260619 14:46:47.007830 4290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:46.991598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42045 (local address 127.0.98.2:45797)
0619 14:46:46.992152 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:46.992199 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:46.993591 (+ 1392us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:46.994234 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:46.994242 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:46.994733 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:46:46.995444 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:46.995470 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:46.996315 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:46.996319 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:46.997550 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:46.997565 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:46.998149 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:46.998156 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:46.998399 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:46.999124 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:46:46.999151 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:46:47.001584 (+ 2433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:46:47.004121 (+ 2537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:47.004128 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:47.004143 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:47.004535 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:47.004974 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:47.004977 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:47.004979 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:47.005194 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:47.005663 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:46:47.005672 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:46:47.005970 (+ 298us) client_negotiation.cc:769] Sending connection context
0619 14:46:47.006179 (+ 209us) client_negotiation.cc:240] Negotiation successful
0619 14:46:47.006549 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":129,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260619 14:46:47.009588 4288 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42045
I20260619 14:46:47.009912 4288 heartbeater.cc:464] Registering TS with master...
I20260619 14:46:47.010507 4288 heartbeater.cc:511] Master 127.0.98.62:42045 requested a full tablet report, sending...
I20260619 14:46:47.011907 3963 ts_manager.cc:194] Registered new tserver with Master: 12ee725b38344d839926962afefff5b6 (127.0.98.2:34279)
I20260619 14:46:47.013052 3963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:45797
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880383132139-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880383132139-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 14:46:47.030292 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880383132139-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880383132139-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880383132139-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880383132139-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42045
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880383132139-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:46:47.159521 4293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:46:47.159834 4293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:46:47.159883 4293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:46:47.164783 4293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:46:47.164886 4293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:46:47.164983 4293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 14:46:47.171110 4293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880383132139-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880383132139-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880383132139-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880383132139-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42045
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4293
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880383132139-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:46:47.172281 4293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:46:47.173436 4293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:46:47.185963 4301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:47.185963 4298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:47.186017 4299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:47.186309 4293 server_base.cc:1034] running on GCE node
I20260619 14:46:47.186966 4293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:46:47.187726 4293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:46:47.188939 4293 hybrid_clock.cc:648] HybridClock initialized: now 1781880407188904 us; error 49 us; skew 500 ppm
Jun 19 14:46:47 dist-test-slave-2sb1 krb5kdc[3918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880407, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:46:47.192101 4293 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 14:46:47.193188 4293 webserver.cc:460] Webserver started at http://127.0.98.3:41353/ using document root <none> and password file <none>
I20260619 14:46:47.193789 4293 fs_manager.cc:359] Metadata directory not provided
I20260619 14:46:47.193897 4293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:46:47.194136 4293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:46:47.196316 4293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880383132139-392-0/minicluster-data/ts-2/data/instance:
uuid: "b4ce7fe8988d4c35b48c9743c6ee8b81"
format_stamp: "Formatted at 2026-06-19 14:46:47 on dist-test-slave-2sb1"
server_key: "2780bcdf550d2b2ef1869b25360d558c"
server_key_iv: "91e0ce0921725fb90c31b4c70afa8e53"
server_key_version: "encryptionkey@0"
I20260619 14:46:47.196978 4293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880383132139-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b4ce7fe8988d4c35b48c9743c6ee8b81"
format_stamp: "Formatted at 2026-06-19 14:46:47 on dist-test-slave-2sb1"
server_key: "2780bcdf550d2b2ef1869b25360d558c"
server_key_iv: "91e0ce0921725fb90c31b4c70afa8e53"
server_key_version: "encryptionkey@0"
I20260619 14:46:47.201319 4293 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 14:46:47.204293 4308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:47.206045 4293 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:46:47.206229 4293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880383132139-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880383132139-392-0/minicluster-data/ts-2/wal
uuid: "b4ce7fe8988d4c35b48c9743c6ee8b81"
format_stamp: "Formatted at 2026-06-19 14:46:47 on dist-test-slave-2sb1"
server_key: "2780bcdf550d2b2ef1869b25360d558c"
server_key_iv: "91e0ce0921725fb90c31b4c70afa8e53"
server_key_version: "encryptionkey@0"
I20260619 14:46:47.206382 4293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880383132139-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880383132139-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880383132139-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:46:47.245078 4293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:46:47.249687 4293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:46:47.249933 4293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:46:47.250643 4293 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:46:47.251592 4293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:46:47.251647 4293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:47.251724 4293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:46:47.251765 4293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:47.262827 4293 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45585
I20260619 14:46:47.262969 4421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45585 every 8 connection(s)
I20260619 14:46:47.263950 4293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880383132139-392-0/minicluster-data/ts-2/data/info.pb
I20260619 14:46:47.269770 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 4293
I20260619 14:46:47.269938 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781880383132139-392-0/minicluster-data/ts-2/wal/instance
I20260619 14:46:47.270242 392 external_mini_cluster.cc:1428] Setting key 0daa96f57f270104dbacb10f1c277fa6
Jun 19 14:46:47 dist-test-slave-2sb1 krb5kdc[3918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880407, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:46:47.281693 4002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:47.266345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57961 (local address 127.0.98.62:42045)
0619 14:46:47.266644 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:47.266648 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:47.267433 (+ 785us) server_negotiation.cc:407] Connection header received
0619 14:46:47.269121 (+ 1688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:47.269124 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:47.269177 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:47.269271 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:47.271044 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:47.271634 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:47.273585 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:47.273809 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:47.273933 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:47.277246 (+ 3313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:47.277250 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:47.277252 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:47.277281 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:47.279089 (+ 1808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:47.279746 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:47.279749 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:47.279750 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:47.279836 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:47.280296 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:47.280299 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:47.280300 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:47.280521 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:47.280608 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:47.281418 (+ 810us) server_negotiation.cc:299] Negotiation successful
0619 14:46:47.281557 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203}
I20260619 14:46:47.282832 4424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:47.266726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42045 (local address 127.0.98.3:57961)
0619 14:46:47.267246 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:47.267293 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:47.268879 (+ 1586us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:47.269665 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:47.269676 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:47.270274 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:46:47.270885 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:47.270901 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:47.271795 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:47.271800 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:47.273448 (+ 1648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:47.273460 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:47.274048 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:47.274052 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:47.274294 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:47.275116 (+ 822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:46:47.275135 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:46:47.277044 (+ 1909us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:46:47.279240 (+ 2196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:47.279247 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:47.279261 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:47.279617 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:47.279977 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:47.279979 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:47.279981 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:47.280143 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:47.280777 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:46:47.280786 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:46:47.281098 (+ 312us) client_negotiation.cc:769] Sending connection context
0619 14:46:47.281307 (+ 209us) client_negotiation.cc:240] Negotiation successful
0619 14:46:47.281615 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":124,"threads_started":1}
I20260619 14:46:47.284623 4422 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42045
I20260619 14:46:47.285015 4422 heartbeater.cc:464] Registering TS with master...
I20260619 14:46:47.285697 4422 heartbeater.cc:511] Master 127.0.98.62:42045 requested a full tablet report, sending...
I20260619 14:46:47.287039 3963 ts_manager.cc:194] Registered new tserver with Master: b4ce7fe8988d4c35b48c9743c6ee8b81 (127.0.98.3:45585)
I20260619 14:46:47.287639 3963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57961
I20260619 14:46:47.298084 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 19 14:46:47 dist-test-slave-2sb1 krb5kdc[3918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880407, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 14:46:47 dist-test-slave-2sb1 krb5kdc[3918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880407, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 14:46:47.330654 4429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:47.317282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56020 (local address 127.0.98.1:34741)
0619 14:46:47.317561 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:47.317570 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:47.317607 (+ 37us) server_negotiation.cc:407] Connection header received
0619 14:46:47.317658 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:47.317663 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:47.317904 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:47.318049 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:47.319143 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:47.320203 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:47.321539 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:47.322012 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:47.322117 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:47.325971 (+ 3854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:47.325985 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:47.326002 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:47.326036 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:47.328090 (+ 2054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:47.328617 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:47.328624 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:47.328625 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:47.328792 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:47.329202 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:47.329205 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:47.329207 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:47.329534 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:47.329725 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:47.330082 (+ 357us) server_negotiation.cc:299] Negotiation successful
0619 14:46:47.330320 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":101,"threads_started":1}
Jun 19 14:46:47 dist-test-slave-2sb1 krb5kdc[3918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880407, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:46:47.345032 4002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:47.334213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59306 (local address 127.0.98.62:42045)
0619 14:46:47.334481 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:47.334485 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:47.334551 (+ 66us) server_negotiation.cc:407] Connection header received
0619 14:46:47.334720 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:47.334723 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:47.334771 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:47.334860 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:47.335665 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:47.336216 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:47.337230 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:47.337598 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:47.337696 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:47.340789 (+ 3093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:47.340793 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:47.340796 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:47.340830 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:47.342753 (+ 1923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:47.343246 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:47.343249 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:47.343251 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:47.343471 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:47.343821 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:47.343825 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:47.343827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:47.344074 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:47.344196 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:47.344757 (+ 561us) server_negotiation.cc:299] Negotiation successful
0619 14:46:47.344900 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191}
W20260619 14:46:47.345616 3963 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59306
I20260619 14:46:47.346573 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 4025
I20260619 14:46:47.352222 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 4159
I20260619 14:46:47.358362 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 4293
I20260619 14:46:47.364141 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 3931
2026-06-19T14:46:47Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2454 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880383132139-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:46:47 dist-test-slave-2sb1 krb5kdc[4434](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:46:47 dist-test-slave-2sb1 krb5kdc[4434](info): set up 2 sockets
Jun 19 14:46:47 dist-test-slave-2sb1 krb5kdc[4434](info): commencing operation
krb5kdc: starting...
W20260619 14:46:48.517253 392 mini_kdc.cc:121] Time spent starting KDC: real 1.130s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880383132139-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880383132139-392-0/krb5kdc/test-user.keytab.
Jun 19 14:46:48 dist-test-slave-2sb1 krb5kdc[4434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:46:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:46:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 14:46:48.676020 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880383132139-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880383132139-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880383132139-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32889
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32889
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:46:48.795578 4447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:46:48.795894 4447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:46:48.795941 4447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:46:48.800170 4447 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:46:48.800266 4447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:46:48.800287 4447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:46:48.800303 4447 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:46:48.800319 4447 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:46:48.806746 4447 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880383132139-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880383132139-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32889
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32889
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4447
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880383132139-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:46:48.807920 4447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:46:48.808992 4447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 14:46:48.818836 4447 server_base.cc:1034] running on GCE node
W20260619 14:46:48.818877 4455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:48.819020 4452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:48.819070 4453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:48.819789 4447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:46:48.821061 4447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:46:48.822280 4447 hybrid_clock.cc:648] HybridClock initialized: now 1781880408822248 us; error 51 us; skew 500 ppm
Jun 19 14:46:48 dist-test-slave-2sb1 krb5kdc[4434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880408, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:46:48.825788 4447 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 14:46:48.826701 4447 fs_manager.cc:359] Metadata directory not provided
I20260619 14:46:48.826812 4447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:46:48.827064 4447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:46:48.829341 4447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880383132139-392-0/minicluster-data/master-0/data/instance:
uuid: "ba356c35c2cb4fd38fc9830b168cc10a"
format_stamp: "Formatted at 2026-06-19 14:46:48 on dist-test-slave-2sb1"
server_key: "bb74b0d83bae8b05e5544087706859ae"
server_key_iv: "16c9f2f8dc66c7bf530aefac7bf60e76"
server_key_version: "encryptionkey@0"
I20260619 14:46:48.829957 4447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880383132139-392-0/minicluster-data/master-0/wal/instance:
uuid: "ba356c35c2cb4fd38fc9830b168cc10a"
format_stamp: "Formatted at 2026-06-19 14:46:48 on dist-test-slave-2sb1"
server_key: "bb74b0d83bae8b05e5544087706859ae"
server_key_iv: "16c9f2f8dc66c7bf530aefac7bf60e76"
server_key_version: "encryptionkey@0"
I20260619 14:46:48.834116 4447 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260619 14:46:48.837091 4461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:48.838636 4447 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 14:46:48.838831 4447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880383132139-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880383132139-392-0/minicluster-data/master-0/wal
uuid: "ba356c35c2cb4fd38fc9830b168cc10a"
format_stamp: "Formatted at 2026-06-19 14:46:48 on dist-test-slave-2sb1"
server_key: "bb74b0d83bae8b05e5544087706859ae"
server_key_iv: "16c9f2f8dc66c7bf530aefac7bf60e76"
server_key_version: "encryptionkey@0"
I20260619 14:46:48.838984 4447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880383132139-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880383132139-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880383132139-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:46:48.854362 4447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:46:48.858515 4447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:46:48.858757 4447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:46:48.866410 4447 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:32889
I20260619 14:46:48.866520 4513 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:32889 every 8 connection(s)
I20260619 14:46:48.867182 4447 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:46:48.870322 4514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:46:48.873817 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 4447
I20260619 14:46:48.873935 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880383132139-392-0/minicluster-data/master-0/wal/instance
I20260619 14:46:48.874248 392 external_mini_cluster.cc:1428] Setting key 915e9af21184a12fcf7e6aad5a427384
I20260619 14:46:48.878317 4514 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba356c35c2cb4fd38fc9830b168cc10a: Bootstrap starting.
I20260619 14:46:48.881604 4514 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba356c35c2cb4fd38fc9830b168cc10a: Neither blocks nor log segments found. Creating new log.
I20260619 14:46:48.882851 4514 log.cc:824] T 00000000000000000000000000000000 P ba356c35c2cb4fd38fc9830b168cc10a: Log is configured to *not* fsync() on all Append() calls
I20260619 14:46:48.885748 4514 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba356c35c2cb4fd38fc9830b168cc10a: No bootstrap required, opened a new log
Jun 19 14:46:48 dist-test-slave-2sb1 krb5kdc[4434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:46:48.889950 4514 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba356c35c2cb4fd38fc9830b168cc10a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba356c35c2cb4fd38fc9830b168cc10a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32889 } }
I20260619 14:46:48.890206 4514 raft_consensus.cc:374] T 00000000000000000000000000000000 P ba356c35c2cb4fd38fc9830b168cc10a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:46:48.890266 4514 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba356c35c2cb4fd38fc9830b168cc10a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba356c35c2cb4fd38fc9830b168cc10a, State: Initialized, Role: FOLLOWER
I20260619 14:46:48.890844 4514 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba356c35c2cb4fd38fc9830b168cc10a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba356c35c2cb4fd38fc9830b168cc10a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32889 } }
I20260619 14:46:48.891036 4514 raft_consensus.cc:388] T 00000000000000000000000000000000 P ba356c35c2cb4fd38fc9830b168cc10a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:46:48.891122 4514 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba356c35c2cb4fd38fc9830b168cc10a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:46:48.891229 4514 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba356c35c2cb4fd38fc9830b168cc10a [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:46:48.892369 4514 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba356c35c2cb4fd38fc9830b168cc10a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba356c35c2cb4fd38fc9830b168cc10a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32889 } }
I20260619 14:46:48.892800 4514 leader_election.cc:302] T 00000000000000000000000000000000 P ba356c35c2cb4fd38fc9830b168cc10a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba356c35c2cb4fd38fc9830b168cc10a; no voters:
I20260619 14:46:48.893365 4514 leader_election.cc:288] T 00000000000000000000000000000000 P ba356c35c2cb4fd38fc9830b168cc10a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:46:48.893494 4519 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba356c35c2cb4fd38fc9830b168cc10a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:46:48.893823 4519 raft_consensus.cc:686] T 00000000000000000000000000000000 P ba356c35c2cb4fd38fc9830b168cc10a [term 1 LEADER]: Becoming Leader. State: Replica: ba356c35c2cb4fd38fc9830b168cc10a, State: Running, Role: LEADER
I20260619 14:46:48.894203 4519 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba356c35c2cb4fd38fc9830b168cc10a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba356c35c2cb4fd38fc9830b168cc10a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32889 } }
I20260619 14:46:48.895135 4514 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba356c35c2cb4fd38fc9830b168cc10a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:46:48.896286 4521 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba356c35c2cb4fd38fc9830b168cc10a [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba356c35c2cb4fd38fc9830b168cc10a. Latest consensus state: current_term: 1 leader_uuid: "ba356c35c2cb4fd38fc9830b168cc10a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba356c35c2cb4fd38fc9830b168cc10a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32889 } } }
I20260619 14:46:48.896437 4521 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba356c35c2cb4fd38fc9830b168cc10a [sys.catalog]: This master's current role is: LEADER
I20260619 14:46:48.896670 4517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:48.876051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38456 (local address 127.0.98.62:32889)
0619 14:46:48.876593 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:48.876625 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:48.876697 (+ 72us) server_negotiation.cc:407] Connection header received
0619 14:46:48.877659 (+ 962us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:48.877686 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:48.878209 (+ 523us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:48.878655 (+ 446us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:48.879570 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:48.881369 (+ 1799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:48.882816 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:48.883661 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:48.883808 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:48.890420 (+ 6612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:48.890426 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:48.890449 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:48.890483 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:48.892871 (+ 2388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:48.893531 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:48.893538 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:48.893545 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:48.893709 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:48.894101 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:48.894104 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:48.894106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:48.894637 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:48.894830 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:48.895170 (+ 340us) server_negotiation.cc:299] Negotiation successful
0619 14:46:48.895502 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":162,"threads_started":1}
I20260619 14:46:48.896701 4521 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba356c35c2cb4fd38fc9830b168cc10a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ba356c35c2cb4fd38fc9830b168cc10a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba356c35c2cb4fd38fc9830b168cc10a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32889 } } }
I20260619 14:46:48.896782 4521 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba356c35c2cb4fd38fc9830b168cc10a [sys.catalog]: This master's current role is: LEADER
W20260619 14:46:48.900638 4534 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ba356c35c2cb4fd38fc9830b168cc10a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:46:48.900820 4534 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:46:48.900906 4528 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:46:48.901998 4528 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:46:48.908504 4528 catalog_manager.cc:1347] Generated new cluster ID: 1c3d7a194ecd4fafb2d7824bfea3f68c
I20260619 14:46:48.908609 4528 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:46:48.940412 4528 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:46:48.941824 4528 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:46:48.952790 4528 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ba356c35c2cb4fd38fc9830b168cc10a: Generated new TSK 0
I20260619 14:46:48.953724 4528 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880383132139-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880383132139-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 14:46:48.992861 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880383132139-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880383132139-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880383132139-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880383132139-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32889
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880383132139-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:46:49.126906 4540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:46:49.127216 4540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:46:49.127264 4540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:46:49.131868 4540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:46:49.131978 4540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:46:49.132052 4540 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 14:46:49.138203 4540 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880383132139-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880383132139-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880383132139-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880383132139-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32889
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4540
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880383132139-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:46:49.139497 4540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:46:49.140628 4540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:46:49.152097 4548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:49.152092 4546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:49.152092 4545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:49.152700 4540 server_base.cc:1034] running on GCE node
I20260619 14:46:49.153165 4540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:46:49.153908 4540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:46:49.155107 4540 hybrid_clock.cc:648] HybridClock initialized: now 1781880409155010 us; error 111 us; skew 500 ppm
Jun 19 14:46:49 dist-test-slave-2sb1 krb5kdc[4434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880409, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:46:49.158751 4540 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 14:46:49.159719 4540 fs_manager.cc:359] Metadata directory not provided
I20260619 14:46:49.159795 4540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:46:49.160065 4540 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:46:49.162442 4540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880383132139-392-0/minicluster-data/ts-0/data/instance:
uuid: "e85f623b78bd484489fc27f261be42ed"
format_stamp: "Formatted at 2026-06-19 14:46:49 on dist-test-slave-2sb1"
server_key: "b90c42441b9dc7c2da2662a8444a004a"
server_key_iv: "31d044ac6dfabc6038646297ab4c39ce"
server_key_version: "encryptionkey@0"
I20260619 14:46:49.163214 4540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880383132139-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e85f623b78bd484489fc27f261be42ed"
format_stamp: "Formatted at 2026-06-19 14:46:49 on dist-test-slave-2sb1"
server_key: "b90c42441b9dc7c2da2662a8444a004a"
server_key_iv: "31d044ac6dfabc6038646297ab4c39ce"
server_key_version: "encryptionkey@0"
I20260619 14:46:49.167933 4540 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260619 14:46:49.171039 4554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:49.172557 4540 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 14:46:49.172773 4540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880383132139-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880383132139-392-0/minicluster-data/ts-0/wal
uuid: "e85f623b78bd484489fc27f261be42ed"
format_stamp: "Formatted at 2026-06-19 14:46:49 on dist-test-slave-2sb1"
server_key: "b90c42441b9dc7c2da2662a8444a004a"
server_key_iv: "31d044ac6dfabc6038646297ab4c39ce"
server_key_version: "encryptionkey@0"
I20260619 14:46:49.172967 4540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880383132139-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880383132139-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880383132139-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:46:49.191802 4540 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:46:49.195997 4540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:46:49.196252 4540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:46:49.197132 4540 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:46:49.198065 4540 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:46:49.198138 4540 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:49.198207 4540 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:46:49.198237 4540 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:49.207926 4540 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34845
I20260619 14:46:49.207976 4667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34845 every 8 connection(s)
I20260619 14:46:49.208595 4540 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880383132139-392-0/minicluster-data/ts-0/data/info.pb
I20260619 14:46:49.211064 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 4540
I20260619 14:46:49.211166 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880383132139-392-0/minicluster-data/ts-0/wal/instance
I20260619 14:46:49.211537 392 external_mini_cluster.cc:1428] Setting key 9326686e31b7ede8f00c48826e602a60
Jun 19 14:46:49 dist-test-slave-2sb1 krb5kdc[4434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880409, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 14:46:49.227102 4517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:49.210482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43701 (local address 127.0.98.62:32889)
0619 14:46:49.210646 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:49.210650 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:49.211738 (+ 1088us) server_negotiation.cc:407] Connection header received
0619 14:46:49.213637 (+ 1899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:49.213640 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:49.213683 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:49.213753 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:49.215505 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:49.216400 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:49.217910 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:49.218347 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:49.218502 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:49.222016 (+ 3514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:49.222020 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:49.222023 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:49.222060 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:49.223803 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:49.224630 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:49.224634 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:49.224635 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:49.224801 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:49.225409 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:49.225412 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:49.225414 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:49.225707 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:49.225816 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:49.226810 (+ 994us) server_negotiation.cc:299] Negotiation successful
0619 14:46:49.226965 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 14:46:49.228279 4670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:49.210929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32889 (local address 127.0.98.1:43701)
0619 14:46:49.211547 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:49.211608 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:49.213233 (+ 1625us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:49.213971 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:49.213984 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:49.214575 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:46:49.215328 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:49.215347 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:49.216557 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:49.216560 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:49.217768 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:49.217778 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:49.218739 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:49.218742 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:49.218927 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:49.219735 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:46:49.219753 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:46:49.221746 (+ 1993us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:46:49.224024 (+ 2278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:49.224030 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:49.224046 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:49.224473 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:49.224940 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:49.224943 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:49.224945 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:49.225141 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:49.225849 (+ 708us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:46:49.225856 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:46:49.226438 (+ 582us) client_negotiation.cc:769] Sending connection context
0619 14:46:49.226712 (+ 274us) client_negotiation.cc:240] Negotiation successful
0619 14:46:49.227164 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":154,"threads_started":1}
I20260619 14:46:49.230113 4668 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32889
I20260619 14:46:49.230609 4668 heartbeater.cc:464] Registering TS with master...
I20260619 14:46:49.231200 4668 heartbeater.cc:511] Master 127.0.98.62:32889 requested a full tablet report, sending...
I20260619 14:46:49.233618 4478 ts_manager.cc:194] Registered new tserver with Master: e85f623b78bd484489fc27f261be42ed (127.0.98.1:34845)
I20260619 14:46:49.235481 4478 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:43701
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880383132139-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880383132139-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 14:46:49.252462 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880383132139-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880383132139-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880383132139-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880383132139-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32889
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880383132139-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:46:49.378932 4673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:46:49.379252 4673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:46:49.379302 4673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:46:49.383694 4673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:46:49.383780 4673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:46:49.383846 4673 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 14:46:49.390080 4673 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880383132139-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880383132139-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880383132139-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880383132139-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32889
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4673
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880383132139-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:46:49.391199 4673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:46:49.392236 4673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:46:49.402901 4678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:49.402897 4681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:49.403024 4679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:49.403142 4673 server_base.cc:1034] running on GCE node
I20260619 14:46:49.403705 4673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:46:49.404543 4673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:46:49.405738 4673 hybrid_clock.cc:648] HybridClock initialized: now 1781880409405724 us; error 34 us; skew 500 ppm
Jun 19 14:46:49 dist-test-slave-2sb1 krb5kdc[4434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880409, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:46:49.409304 4673 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 14:46:49.410219 4673 fs_manager.cc:359] Metadata directory not provided
I20260619 14:46:49.410293 4673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:46:49.410583 4673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:46:49.412954 4673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880383132139-392-0/minicluster-data/ts-1/data/instance:
uuid: "d4111b368ec74fc6a14eeb9d6c9a6b89"
format_stamp: "Formatted at 2026-06-19 14:46:49 on dist-test-slave-2sb1"
server_key: "67a8c71f80a9370919bf0ffee6c792b0"
server_key_iv: "3529296a0c2fbd913f732abaa032506d"
server_key_version: "encryptionkey@0"
I20260619 14:46:49.413585 4673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880383132139-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d4111b368ec74fc6a14eeb9d6c9a6b89"
format_stamp: "Formatted at 2026-06-19 14:46:49 on dist-test-slave-2sb1"
server_key: "67a8c71f80a9370919bf0ffee6c792b0"
server_key_iv: "3529296a0c2fbd913f732abaa032506d"
server_key_version: "encryptionkey@0"
I20260619 14:46:49.418020 4673 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 14:46:49.422025 4687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:49.423642 4673 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260619 14:46:49.423869 4673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880383132139-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880383132139-392-0/minicluster-data/ts-1/wal
uuid: "d4111b368ec74fc6a14eeb9d6c9a6b89"
format_stamp: "Formatted at 2026-06-19 14:46:49 on dist-test-slave-2sb1"
server_key: "67a8c71f80a9370919bf0ffee6c792b0"
server_key_iv: "3529296a0c2fbd913f732abaa032506d"
server_key_version: "encryptionkey@0"
I20260619 14:46:49.424036 4673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880383132139-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880383132139-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880383132139-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:46:49.440857 4673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:46:49.444615 4673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:46:49.444903 4673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:46:49.445545 4673 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:46:49.446439 4673 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:46:49.446512 4673 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:49.446587 4673 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:46:49.446638 4673 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:49.458014 4673 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43135
I20260619 14:46:49.458034 4800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43135 every 8 connection(s)
I20260619 14:46:49.458683 4673 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880383132139-392-0/minicluster-data/ts-1/data/info.pb
I20260619 14:46:49.461621 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 4673
I20260619 14:46:49.461756 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880383132139-392-0/minicluster-data/ts-1/wal/instance
I20260619 14:46:49.462076 392 external_mini_cluster.cc:1428] Setting key 4d82ed35aa831d23339525d4ccedb89a
Jun 19 14:46:49 dist-test-slave-2sb1 krb5kdc[4434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880409, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260619 14:46:49.475807 4517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:49.461071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37407 (local address 127.0.98.62:32889)
0619 14:46:49.461209 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:49.461213 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:49.462487 (+ 1274us) server_negotiation.cc:407] Connection header received
0619 14:46:49.464178 (+ 1691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:49.464182 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:49.464238 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:49.464341 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:49.465790 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:49.466326 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:49.467614 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:49.467849 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:49.467967 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:49.471257 (+ 3290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:49.471261 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:49.471264 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:49.471296 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:49.473270 (+ 1974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:49.473979 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:49.473989 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:49.473991 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:49.474100 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:49.474471 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:49.474476 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:49.474477 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:49.474678 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:49.474867 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:49.475499 (+ 632us) server_negotiation.cc:299] Negotiation successful
0619 14:46:49.475638 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 14:46:49.476888 4803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:49.461485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32889 (local address 127.0.98.2:37407)
0619 14:46:49.462198 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:49.462253 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:49.463886 (+ 1633us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:49.464553 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:49.464561 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:49.465015 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:46:49.465621 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:49.465637 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:49.466469 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:49.466473 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:49.467470 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:49.467480 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:49.468027 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:49.468031 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:49.468209 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:49.469005 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:46:49.469030 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:46:49.471039 (+ 2009us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:46:49.473431 (+ 2392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:49.473442 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:49.473476 (+ 34us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:49.473860 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:49.474203 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:49.474206 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:49.474208 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:49.474370 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:49.474855 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:46:49.474862 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:46:49.475194 (+ 332us) client_negotiation.cc:769] Sending connection context
0619 14:46:49.475461 (+ 267us) client_negotiation.cc:240] Negotiation successful
0619 14:46:49.475825 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":169,"threads_started":1}
I20260619 14:46:49.478271 4801 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32889
I20260619 14:46:49.478570 4801 heartbeater.cc:464] Registering TS with master...
I20260619 14:46:49.479169 4801 heartbeater.cc:511] Master 127.0.98.62:32889 requested a full tablet report, sending...
I20260619 14:46:49.480719 4478 ts_manager.cc:194] Registered new tserver with Master: d4111b368ec74fc6a14eeb9d6c9a6b89 (127.0.98.2:43135)
I20260619 14:46:49.481771 4478 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:37407
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880383132139-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880383132139-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 14:46:49.498093 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880383132139-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880383132139-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880383132139-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880383132139-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32889
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880383132139-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:46:49.624557 4806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:46:49.624972 4806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:46:49.625028 4806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:46:49.629963 4806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:46:49.630059 4806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:46:49.630126 4806 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 14:46:49.635843 4806 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880383132139-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880383132139-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880383132139-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880383132139-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32889
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4806
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880383132139-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:46:49.637001 4806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:46:49.638006 4806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:46:49.648140 4811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:49.648139 4812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:49.648145 4814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:49.648181 4806 server_base.cc:1034] running on GCE node
I20260619 14:46:49.648929 4806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:46:49.649638 4806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:46:49.650817 4806 hybrid_clock.cc:648] HybridClock initialized: now 1781880409650782 us; error 49 us; skew 500 ppm
Jun 19 14:46:49 dist-test-slave-2sb1 krb5kdc[4434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880409, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:46:49.654023 4806 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 14:46:49.654991 4806 fs_manager.cc:359] Metadata directory not provided
I20260619 14:46:49.655102 4806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:46:49.655360 4806 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:46:49.657737 4806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880383132139-392-0/minicluster-data/ts-2/data/instance:
uuid: "d9d1cf421c6049dbbcf226700b89f7ae"
format_stamp: "Formatted at 2026-06-19 14:46:49 on dist-test-slave-2sb1"
server_key: "202ae55e7e94c660b041dd526320881a"
server_key_iv: "90bfb138c28be786b1ea2eb224a2fa76"
server_key_version: "encryptionkey@0"
I20260619 14:46:49.658340 4806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880383132139-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d9d1cf421c6049dbbcf226700b89f7ae"
format_stamp: "Formatted at 2026-06-19 14:46:49 on dist-test-slave-2sb1"
server_key: "202ae55e7e94c660b041dd526320881a"
server_key_iv: "90bfb138c28be786b1ea2eb224a2fa76"
server_key_version: "encryptionkey@0"
I20260619 14:46:49.662405 4806 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 14:46:49.665325 4820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:49.666844 4806 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:46:49.667004 4806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880383132139-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880383132139-392-0/minicluster-data/ts-2/wal
uuid: "d9d1cf421c6049dbbcf226700b89f7ae"
format_stamp: "Formatted at 2026-06-19 14:46:49 on dist-test-slave-2sb1"
server_key: "202ae55e7e94c660b041dd526320881a"
server_key_iv: "90bfb138c28be786b1ea2eb224a2fa76"
server_key_version: "encryptionkey@0"
I20260619 14:46:49.667150 4806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880383132139-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880383132139-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880383132139-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:46:49.690480 4806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:46:49.694578 4806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:46:49.694828 4806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:46:49.695590 4806 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:46:49.696904 4806 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:46:49.697005 4806 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:49.697103 4806 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:46:49.697163 4806 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:49.709318 4806 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34851
I20260619 14:46:49.709450 4933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34851 every 8 connection(s)
I20260619 14:46:49.710136 4806 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880383132139-392-0/minicluster-data/ts-2/data/info.pb
I20260619 14:46:49.716903 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 4806
I20260619 14:46:49.717031 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880383132139-392-0/minicluster-data/ts-2/wal/instance
I20260619 14:46:49.717567 392 external_mini_cluster.cc:1428] Setting key 0a00cf7454beec4a9a6bf778490aa230
Jun 19 14:46:49 dist-test-slave-2sb1 krb5kdc[4434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880409, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:46:49.728322 4517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:49.712369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36045 (local address 127.0.98.62:32889)
0619 14:46:49.712531 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:49.712535 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:49.713784 (+ 1249us) server_negotiation.cc:407] Connection header received
0619 14:46:49.715288 (+ 1504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:49.715291 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:49.715344 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:49.715438 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:49.717344 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:49.718258 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:49.720080 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:49.720477 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:49.720615 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:49.724024 (+ 3409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:49.724027 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:49.724030 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:49.724057 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:49.725692 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:49.726464 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:49.726476 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:49.726477 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:49.726562 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:49.727006 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:49.727008 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:49.727009 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:49.727230 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:49.727303 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:49.727955 (+ 652us) server_negotiation.cc:299] Negotiation successful
0619 14:46:49.728113 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 14:46:49.729589 4936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:49.712890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32889 (local address 127.0.98.3:36045)
0619 14:46:49.713604 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:49.713652 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:49.715035 (+ 1383us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:49.715749 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:49.715758 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:49.716265 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:46:49.717159 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:49.717180 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:49.718537 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:49.718540 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:49.719929 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:49.719941 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:49.720803 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:49.720807 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:49.720997 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:49.721809 (+ 812us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:46:49.721830 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:46:49.723785 (+ 1955us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:46:49.725831 (+ 2046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:49.725838 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:49.725854 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:49.726229 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:49.726662 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:49.726665 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:49.726667 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:49.726827 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:49.727340 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:46:49.727347 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:46:49.727656 (+ 309us) client_negotiation.cc:769] Sending connection context
0619 14:46:49.727966 (+ 310us) client_negotiation.cc:240] Negotiation successful
0619 14:46:49.728479 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":153,"threads_started":1}
I20260619 14:46:49.731086 4934 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32889
I20260619 14:46:49.731386 4934 heartbeater.cc:464] Registering TS with master...
I20260619 14:46:49.732013 4934 heartbeater.cc:511] Master 127.0.98.62:32889 requested a full tablet report, sending...
I20260619 14:46:49.733263 4478 ts_manager.cc:194] Registered new tserver with Master: d9d1cf421c6049dbbcf226700b89f7ae (127.0.98.3:34851)
I20260619 14:46:49.733963 4478 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:36045
I20260619 14:46:49.745508 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:46:49.755638 4517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:49.747716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38470 (local address 127.0.98.62:32889)
0619 14:46:49.747875 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:49.747880 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:49.747893 (+ 13us) server_negotiation.cc:407] Connection header received
0619 14:46:49.747937 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:49.747939 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:49.747977 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:49.748067 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:49.748902 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:49.749431 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:49.750303 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:49.750533 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:49.750654 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:49.752176 (+ 1522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:49.752180 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:49.752182 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:49.752209 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:49.753739 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:49.754338 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:49.754341 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:49.754343 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:49.754421 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:49.754701 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:49.754706 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:49.754708 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:49.754950 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:49.755097 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:49.755342 (+ 245us) server_negotiation.cc:299] Negotiation successful
0619 14:46:49.755463 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 14:46:49.759423 4478 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38470:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:46:49.762923 4478 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:46:49.779119 4950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:49.773943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52792 (local address 127.0.98.1:34845)
0619 14:46:49.774258 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:49.774265 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:49.774284 (+ 19us) server_negotiation.cc:407] Connection header received
0619 14:46:49.774346 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:49.774350 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:49.774491 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:49.775019 (+ 528us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:46:49.775166 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:49.776015 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:49.777725 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:49.778399 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:49.778545 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:49.778704 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:49.778848 (+ 144us) server_negotiation.cc:299] Negotiation successful
0619 14:46:49.778955 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":68,"threads_started":1}
I20260619 14:46:49.779111 4945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:49.773227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34845 (local address 127.0.0.1:52792)
0619 14:46:49.773692 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:49.773747 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:49.774150 (+ 403us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:49.774655 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:49.774664 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:49.774685 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:46:49.775018 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:49.775027 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:49.776135 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:49.776142 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:49.777589 (+ 1447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:49.777599 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:49.778542 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:49.778545 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:49.778658 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:49.778695 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 14:46:49.778816 (+ 121us) client_negotiation.cc:240] Negotiation successful
0619 14:46:49.778946 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"mutex_wait_us":50,"spinlock_wait_cycles":857600,"thread_start_us":114,"threads_started":1}
I20260619 14:46:49.780130 4947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:49.772726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34851 (local address 127.0.0.1:38806)
0619 14:46:49.773690 (+ 964us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:49.773746 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:49.773932 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:49.774682 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:49.774685 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:49.774706 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:46:49.774944 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:49.774953 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:49.776943 (+ 1990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:49.776945 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:49.778578 (+ 1633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:49.778588 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:49.779751 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:49.779755 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:49.779852 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:49.779875 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 14:46:49.779939 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 14:46:49.780004 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":700,"mutex_wait_us":397,"thread_start_us":118,"threads_started":1}
I20260619 14:46:49.780253 4948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:49.773531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43135 (local address 127.0.0.1:37592)
0619 14:46:49.774380 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:49.774395 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:49.774478 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:49.775234 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:49.775237 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:49.775253 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:46:49.775337 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:49.775343 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:49.776918 (+ 1575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:49.776923 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:49.778357 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:49.778370 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:49.779792 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:49.779796 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:49.779903 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:49.779929 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 14:46:49.779988 (+ 59us) client_negotiation.cc:240] Negotiation successful
0619 14:46:49.780061 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":776,"mutex_wait_us":57,"thread_start_us":84,"threads_started":1}
I20260619 14:46:49.781121 4949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:49.773744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37592 (local address 127.0.98.2:43135)
0619 14:46:49.774740 (+ 996us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:49.774753 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:49.774791 (+ 38us) server_negotiation.cc:407] Connection header received
0619 14:46:49.774875 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:49.774883 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:49.775093 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:49.775249 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:46:49.775447 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:49.776736 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:49.778495 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:49.779638 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:49.780351 (+ 713us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:49.780802 (+ 451us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:49.780856 (+ 54us) server_negotiation.cc:299] Negotiation successful
0619 14:46:49.780955 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":894,"thread_start_us":117,"threads_started":1}
I20260619 14:46:49.780253 4946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:49.772838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38806 (local address 127.0.98.3:34851)
0619 14:46:49.773366 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:49.773381 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:49.774092 (+ 711us) server_negotiation.cc:407] Connection header received
0619 14:46:49.774206 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:49.774211 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:49.774436 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:49.774606 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:46:49.775094 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:49.776560 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:49.778728 (+ 2168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:49.779622 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:49.779761 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:49.779915 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:49.779974 (+ 59us) server_negotiation.cc:299] Negotiation successful
0619 14:46:49.780070 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":174,"threads_started":1}
I20260619 14:46:49.782963 4602 tablet_service.cc:1467] Processing CreateTablet for tablet 322d99ef5dd84b1e943963dc83c0a65e (DEFAULT_TABLE table=test-table [id=e6d98526aeb0474fa6b5d089bcfadbd8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:46:49.784289 4602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 322d99ef5dd84b1e943963dc83c0a65e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:46:49.784152 4868 tablet_service.cc:1467] Processing CreateTablet for tablet 322d99ef5dd84b1e943963dc83c0a65e (DEFAULT_TABLE table=test-table [id=e6d98526aeb0474fa6b5d089bcfadbd8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:46:49.784399 4735 tablet_service.cc:1467] Processing CreateTablet for tablet 322d99ef5dd84b1e943963dc83c0a65e (DEFAULT_TABLE table=test-table [id=e6d98526aeb0474fa6b5d089bcfadbd8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:46:49.785245 4868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 322d99ef5dd84b1e943963dc83c0a65e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:46:49.785499 4735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 322d99ef5dd84b1e943963dc83c0a65e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:46:49.791817 4951 tablet_bootstrap.cc:492] T 322d99ef5dd84b1e943963dc83c0a65e P d9d1cf421c6049dbbcf226700b89f7ae: Bootstrap starting.
I20260619 14:46:49.793345 4952 tablet_bootstrap.cc:492] T 322d99ef5dd84b1e943963dc83c0a65e P e85f623b78bd484489fc27f261be42ed: Bootstrap starting.
I20260619 14:46:49.793745 4953 tablet_bootstrap.cc:492] T 322d99ef5dd84b1e943963dc83c0a65e P d4111b368ec74fc6a14eeb9d6c9a6b89: Bootstrap starting.
I20260619 14:46:49.794477 4951 tablet_bootstrap.cc:654] T 322d99ef5dd84b1e943963dc83c0a65e P d9d1cf421c6049dbbcf226700b89f7ae: Neither blocks nor log segments found. Creating new log.
I20260619 14:46:49.795644 4952 tablet_bootstrap.cc:654] T 322d99ef5dd84b1e943963dc83c0a65e P e85f623b78bd484489fc27f261be42ed: Neither blocks nor log segments found. Creating new log.
I20260619 14:46:49.796123 4951 log.cc:824] T 322d99ef5dd84b1e943963dc83c0a65e P d9d1cf421c6049dbbcf226700b89f7ae: Log is configured to *not* fsync() on all Append() calls
I20260619 14:46:49.796545 4953 tablet_bootstrap.cc:654] T 322d99ef5dd84b1e943963dc83c0a65e P d4111b368ec74fc6a14eeb9d6c9a6b89: Neither blocks nor log segments found. Creating new log.
I20260619 14:46:49.796736 4952 log.cc:824] T 322d99ef5dd84b1e943963dc83c0a65e P e85f623b78bd484489fc27f261be42ed: Log is configured to *not* fsync() on all Append() calls
I20260619 14:46:49.797986 4953 log.cc:824] T 322d99ef5dd84b1e943963dc83c0a65e P d4111b368ec74fc6a14eeb9d6c9a6b89: Log is configured to *not* fsync() on all Append() calls
I20260619 14:46:49.798856 4951 tablet_bootstrap.cc:492] T 322d99ef5dd84b1e943963dc83c0a65e P d9d1cf421c6049dbbcf226700b89f7ae: No bootstrap required, opened a new log
I20260619 14:46:49.798996 4952 tablet_bootstrap.cc:492] T 322d99ef5dd84b1e943963dc83c0a65e P e85f623b78bd484489fc27f261be42ed: No bootstrap required, opened a new log
I20260619 14:46:49.799086 4951 ts_tablet_manager.cc:1397] T 322d99ef5dd84b1e943963dc83c0a65e P d9d1cf421c6049dbbcf226700b89f7ae: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260619 14:46:49.799230 4952 ts_tablet_manager.cc:1397] T 322d99ef5dd84b1e943963dc83c0a65e P e85f623b78bd484489fc27f261be42ed: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260619 14:46:49.800096 4953 tablet_bootstrap.cc:492] T 322d99ef5dd84b1e943963dc83c0a65e P d4111b368ec74fc6a14eeb9d6c9a6b89: No bootstrap required, opened a new log
I20260619 14:46:49.800275 4953 ts_tablet_manager.cc:1397] T 322d99ef5dd84b1e943963dc83c0a65e P d4111b368ec74fc6a14eeb9d6c9a6b89: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260619 14:46:49.803040 4952 raft_consensus.cc:348] T 322d99ef5dd84b1e943963dc83c0a65e P e85f623b78bd484489fc27f261be42ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d1cf421c6049dbbcf226700b89f7ae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34851 } } peers { permanent_uuid: "d4111b368ec74fc6a14eeb9d6c9a6b89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43135 } } peers { permanent_uuid: "e85f623b78bd484489fc27f261be42ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34845 } }
I20260619 14:46:49.803361 4952 raft_consensus.cc:374] T 322d99ef5dd84b1e943963dc83c0a65e P e85f623b78bd484489fc27f261be42ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:46:49.803192 4951 raft_consensus.cc:348] T 322d99ef5dd84b1e943963dc83c0a65e P d9d1cf421c6049dbbcf226700b89f7ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d1cf421c6049dbbcf226700b89f7ae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34851 } } peers { permanent_uuid: "d4111b368ec74fc6a14eeb9d6c9a6b89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43135 } } peers { permanent_uuid: "e85f623b78bd484489fc27f261be42ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34845 } }
I20260619 14:46:49.803674 4951 raft_consensus.cc:374] T 322d99ef5dd84b1e943963dc83c0a65e P d9d1cf421c6049dbbcf226700b89f7ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:46:49.803773 4951 raft_consensus.cc:729] T 322d99ef5dd84b1e943963dc83c0a65e P d9d1cf421c6049dbbcf226700b89f7ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9d1cf421c6049dbbcf226700b89f7ae, State: Initialized, Role: FOLLOWER
I20260619 14:46:49.803443 4952 raft_consensus.cc:729] T 322d99ef5dd84b1e943963dc83c0a65e P e85f623b78bd484489fc27f261be42ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e85f623b78bd484489fc27f261be42ed, State: Initialized, Role: FOLLOWER
I20260619 14:46:49.804425 4951 consensus_queue.cc:260] T 322d99ef5dd84b1e943963dc83c0a65e P d9d1cf421c6049dbbcf226700b89f7ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d1cf421c6049dbbcf226700b89f7ae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34851 } } peers { permanent_uuid: "d4111b368ec74fc6a14eeb9d6c9a6b89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43135 } } peers { permanent_uuid: "e85f623b78bd484489fc27f261be42ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34845 } }
I20260619 14:46:49.804425 4952 consensus_queue.cc:260] T 322d99ef5dd84b1e943963dc83c0a65e P e85f623b78bd484489fc27f261be42ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d1cf421c6049dbbcf226700b89f7ae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34851 } } peers { permanent_uuid: "d4111b368ec74fc6a14eeb9d6c9a6b89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43135 } } peers { permanent_uuid: "e85f623b78bd484489fc27f261be42ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34845 } }
I20260619 14:46:49.804474 4953 raft_consensus.cc:348] T 322d99ef5dd84b1e943963dc83c0a65e P d4111b368ec74fc6a14eeb9d6c9a6b89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d1cf421c6049dbbcf226700b89f7ae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34851 } } peers { permanent_uuid: "d4111b368ec74fc6a14eeb9d6c9a6b89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43135 } } peers { permanent_uuid: "e85f623b78bd484489fc27f261be42ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34845 } }
I20260619 14:46:49.804776 4953 raft_consensus.cc:374] T 322d99ef5dd84b1e943963dc83c0a65e P d4111b368ec74fc6a14eeb9d6c9a6b89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:46:49.804847 4953 raft_consensus.cc:729] T 322d99ef5dd84b1e943963dc83c0a65e P d4111b368ec74fc6a14eeb9d6c9a6b89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4111b368ec74fc6a14eeb9d6c9a6b89, State: Initialized, Role: FOLLOWER
I20260619 14:46:49.805333 4934 heartbeater.cc:503] Master 127.0.98.62:32889 was elected leader, sending a full tablet report...
I20260619 14:46:49.805383 4953 consensus_queue.cc:260] T 322d99ef5dd84b1e943963dc83c0a65e P d4111b368ec74fc6a14eeb9d6c9a6b89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d1cf421c6049dbbcf226700b89f7ae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34851 } } peers { permanent_uuid: "d4111b368ec74fc6a14eeb9d6c9a6b89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43135 } } peers { permanent_uuid: "e85f623b78bd484489fc27f261be42ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34845 } }
I20260619 14:46:49.805727 4951 ts_tablet_manager.cc:1428] T 322d99ef5dd84b1e943963dc83c0a65e P d9d1cf421c6049dbbcf226700b89f7ae: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260619 14:46:49.806083 4801 heartbeater.cc:503] Master 127.0.98.62:32889 was elected leader, sending a full tablet report...
I20260619 14:46:49.806330 4953 ts_tablet_manager.cc:1428] T 322d99ef5dd84b1e943963dc83c0a65e P d4111b368ec74fc6a14eeb9d6c9a6b89: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260619 14:46:49.806890 4952 ts_tablet_manager.cc:1428] T 322d99ef5dd84b1e943963dc83c0a65e P e85f623b78bd484489fc27f261be42ed: Time spent starting tablet: real 0.008s user 0.002s sys 0.004s
I20260619 14:46:49.806897 4668 heartbeater.cc:503] Master 127.0.98.62:32889 was elected leader, sending a full tablet report...
W20260619 14:46:49.960031 4669 tablet.cc:2369] T 322d99ef5dd84b1e943963dc83c0a65e P e85f623b78bd484489fc27f261be42ed: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 14:46:49.960068 4802 tablet.cc:2369] T 322d99ef5dd84b1e943963dc83c0a65e P d4111b368ec74fc6a14eeb9d6c9a6b89: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 14:46:49.961129 4935 tablet.cc:2369] T 322d99ef5dd84b1e943963dc83c0a65e P d9d1cf421c6049dbbcf226700b89f7ae: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:46:50.106854 4957 raft_consensus.cc:482] T 322d99ef5dd84b1e943963dc83c0a65e P e85f623b78bd484489fc27f261be42ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:46:50.107179 4957 raft_consensus.cc:504] T 322d99ef5dd84b1e943963dc83c0a65e P e85f623b78bd484489fc27f261be42ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d1cf421c6049dbbcf226700b89f7ae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34851 } } peers { permanent_uuid: "d4111b368ec74fc6a14eeb9d6c9a6b89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43135 } } peers { permanent_uuid: "e85f623b78bd484489fc27f261be42ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34845 } }
I20260619 14:46:50.108373 4957 leader_election.cc:288] T 322d99ef5dd84b1e943963dc83c0a65e P e85f623b78bd484489fc27f261be42ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9d1cf421c6049dbbcf226700b89f7ae (127.0.98.3:34851), d4111b368ec74fc6a14eeb9d6c9a6b89 (127.0.98.2:43135)
I20260619 14:46:50.109647 4959 raft_consensus.cc:482] T 322d99ef5dd84b1e943963dc83c0a65e P d4111b368ec74fc6a14eeb9d6c9a6b89 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:46:50.109915 4959 raft_consensus.cc:504] T 322d99ef5dd84b1e943963dc83c0a65e P d4111b368ec74fc6a14eeb9d6c9a6b89 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d1cf421c6049dbbcf226700b89f7ae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34851 } } peers { permanent_uuid: "d4111b368ec74fc6a14eeb9d6c9a6b89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43135 } } peers { permanent_uuid: "e85f623b78bd484489fc27f261be42ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34845 } }
I20260619 14:46:50.112798 4959 leader_election.cc:288] T 322d99ef5dd84b1e943963dc83c0a65e P d4111b368ec74fc6a14eeb9d6c9a6b89 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9d1cf421c6049dbbcf226700b89f7ae (127.0.98.3:34851), e85f623b78bd484489fc27f261be42ed (127.0.98.1:34845)
I20260619 14:46:50.115146 4960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:50.108800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34851 (local address 127.0.98.1:41851)
0619 14:46:50.109190 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:50.109210 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:50.109581 (+ 371us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:50.110070 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:50.110072 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:50.110089 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:46:50.110234 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:50.110241 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:50.111048 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:50.111054 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:50.112554 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:50.112568 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:50.114610 (+ 2042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:50.114614 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:50.114731 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:50.114763 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 14:46:50.114850 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 14:46:50.114936 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"mutex_wait_us":58,"thread_start_us":129,"threads_started":1}
I20260619 14:46:50.115460 4949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:50.109393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47549 (local address 127.0.98.2:43135)
0619 14:46:50.109600 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:50.109604 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:50.109739 (+ 135us) server_negotiation.cc:407] Connection header received
0619 14:46:50.109913 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:50.109916 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:50.109970 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:50.110033 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:46:50.110487 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:50.111186 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:50.113612 (+ 2426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:50.114662 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:50.114781 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:50.114858 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:50.115142 (+ 284us) server_negotiation.cc:299] Negotiation successful
0619 14:46:50.115232 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":127}
I20260619 14:46:50.115605 4946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:50.109032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41851 (local address 127.0.98.3:34851)
0619 14:46:50.109205 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:50.109211 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:50.109474 (+ 263us) server_negotiation.cc:407] Connection header received
0619 14:46:50.109804 (+ 330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:50.109807 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:50.109874 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:50.109951 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:46:50.110382 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:50.110926 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:50.112777 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:50.114153 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:50.114463 (+ 310us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:50.114534 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:50.115397 (+ 863us) server_negotiation.cc:299] Negotiation successful
0619 14:46:50.115484 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260619 14:46:50.116326 4961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:50.109230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43135 (local address 127.0.98.1:47549)
0619 14:46:50.109615 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:50.109638 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:50.109779 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:50.110201 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:50.110204 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:50.110228 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:46:50.110354 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:50.110362 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:50.111318 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:50.111321 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:50.112554 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:50.112568 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:50.114915 (+ 2347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:50.114918 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:50.115006 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:50.115026 (+ 20us) client_negotiation.cc:769] Sending connection context
0619 14:46:50.116045 (+ 1019us) client_negotiation.cc:240] Negotiation successful
0619 14:46:50.116138 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"mutex_wait_us":33,"thread_start_us":121,"threads_started":1}
I20260619 14:46:50.116349 4888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "322d99ef5dd84b1e943963dc83c0a65e" candidate_uuid: "e85f623b78bd484489fc27f261be42ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9d1cf421c6049dbbcf226700b89f7ae" is_pre_election: true
I20260619 14:46:50.116730 4888 raft_consensus.cc:2454] T 322d99ef5dd84b1e943963dc83c0a65e P d9d1cf421c6049dbbcf226700b89f7ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e85f623b78bd484489fc27f261be42ed in term 0.
I20260619 14:46:50.117208 4755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "322d99ef5dd84b1e943963dc83c0a65e" candidate_uuid: "e85f623b78bd484489fc27f261be42ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4111b368ec74fc6a14eeb9d6c9a6b89" is_pre_election: true
I20260619 14:46:50.117292 4555 leader_election.cc:302] T 322d99ef5dd84b1e943963dc83c0a65e P e85f623b78bd484489fc27f261be42ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d9d1cf421c6049dbbcf226700b89f7ae, e85f623b78bd484489fc27f261be42ed; no voters:
I20260619 14:46:50.117419 4755 raft_consensus.cc:2454] T 322d99ef5dd84b1e943963dc83c0a65e P d4111b368ec74fc6a14eeb9d6c9a6b89 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e85f623b78bd484489fc27f261be42ed in term 0.
I20260619 14:46:50.117643 4957 raft_consensus.cc:2781] T 322d99ef5dd84b1e943963dc83c0a65e P e85f623b78bd484489fc27f261be42ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:46:50.117735 4957 raft_consensus.cc:482] T 322d99ef5dd84b1e943963dc83c0a65e P e85f623b78bd484489fc27f261be42ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:46:50.117784 4957 raft_consensus.cc:3037] T 322d99ef5dd84b1e943963dc83c0a65e P e85f623b78bd484489fc27f261be42ed [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:46:50.117931 4950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:50.113466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53151 (local address 127.0.98.1:34845)
0619 14:46:50.113585 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:50.113590 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:50.113604 (+ 14us) server_negotiation.cc:407] Connection header received
0619 14:46:50.113644 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:50.113647 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:50.113699 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:50.113771 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:46:50.114142 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:50.115075 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:50.116609 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:50.117231 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:50.117317 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:50.117372 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:50.117731 (+ 359us) server_negotiation.cc:299] Negotiation successful
0619 14:46:50.117829 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 14:46:50.118319 4964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:50.113086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34845 (local address 127.0.98.2:53151)
0619 14:46:50.113365 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:50.113382 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:50.113492 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:50.113893 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:50.113896 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:50.113926 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:46:50.114022 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:50.114028 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:50.115209 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:50.115212 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:50.116438 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:50.116463 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:50.117469 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:50.117472 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:50.117580 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:50.117609 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 14:46:50.118097 (+ 488us) client_negotiation.cc:240] Negotiation successful
0619 14:46:50.118181 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"mutex_wait_us":4,"thread_start_us":85,"threads_started":1}
I20260619 14:46:50.118884 4963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:50.111889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33233 (local address 127.0.98.3:34851)
0619 14:46:50.112579 (+ 690us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:50.112594 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:50.113113 (+ 519us) server_negotiation.cc:407] Connection header received
0619 14:46:50.113164 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:50.113167 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:50.113224 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:50.113314 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:46:50.113775 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:50.114761 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:50.117676 (+ 2915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:50.118279 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:50.118427 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:50.118484 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:50.118681 (+ 197us) server_negotiation.cc:299] Negotiation successful
0619 14:46:50.118755 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":272,"threads_started":1}
I20260619 14:46:50.119009 4962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:50.111806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34851 (local address 127.0.98.2:33233)
0619 14:46:50.112881 (+ 1075us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:50.112896 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:50.113024 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:50.113443 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:50.113446 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:50.113466 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:46:50.113645 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:50.113653 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:50.114905 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:50.114908 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:50.117485 (+ 2577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:50.117496 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:50.118430 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:50.118433 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:50.118537 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:50.118559 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 14:46:50.118665 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 14:46:50.118748 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":983,"mutex_wait_us":409,"spinlock_wait_cycles":6528,"thread_start_us":94,"threads_started":1}
I20260619 14:46:50.119232 4957 raft_consensus.cc:504] T 322d99ef5dd84b1e943963dc83c0a65e P e85f623b78bd484489fc27f261be42ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d1cf421c6049dbbcf226700b89f7ae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34851 } } peers { permanent_uuid: "d4111b368ec74fc6a14eeb9d6c9a6b89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43135 } } peers { permanent_uuid: "e85f623b78bd484489fc27f261be42ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34845 } }
I20260619 14:46:50.119288 4622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "322d99ef5dd84b1e943963dc83c0a65e" candidate_uuid: "d4111b368ec74fc6a14eeb9d6c9a6b89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e85f623b78bd484489fc27f261be42ed" is_pre_election: true
I20260619 14:46:50.119448 4888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "322d99ef5dd84b1e943963dc83c0a65e" candidate_uuid: "d4111b368ec74fc6a14eeb9d6c9a6b89" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9d1cf421c6049dbbcf226700b89f7ae" is_pre_election: true
I20260619 14:46:50.119554 4622 raft_consensus.cc:2379] T 322d99ef5dd84b1e943963dc83c0a65e P e85f623b78bd484489fc27f261be42ed [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d4111b368ec74fc6a14eeb9d6c9a6b89 in current term 1: Already voted for candidate e85f623b78bd484489fc27f261be42ed in this term.
I20260619 14:46:50.119565 4888 raft_consensus.cc:2454] T 322d99ef5dd84b1e943963dc83c0a65e P d9d1cf421c6049dbbcf226700b89f7ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4111b368ec74fc6a14eeb9d6c9a6b89 in term 0.
I20260619 14:46:50.119637 4957 leader_election.cc:288] T 322d99ef5dd84b1e943963dc83c0a65e P e85f623b78bd484489fc27f261be42ed [CANDIDATE]: Term 1 election: Requested vote from peers d9d1cf421c6049dbbcf226700b89f7ae (127.0.98.3:34851), d4111b368ec74fc6a14eeb9d6c9a6b89 (127.0.98.2:43135)
I20260619 14:46:50.119968 4888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "322d99ef5dd84b1e943963dc83c0a65e" candidate_uuid: "e85f623b78bd484489fc27f261be42ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9d1cf421c6049dbbcf226700b89f7ae"
I20260619 14:46:50.120057 4755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "322d99ef5dd84b1e943963dc83c0a65e" candidate_uuid: "e85f623b78bd484489fc27f261be42ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4111b368ec74fc6a14eeb9d6c9a6b89"
I20260619 14:46:50.120086 4888 raft_consensus.cc:3037] T 322d99ef5dd84b1e943963dc83c0a65e P d9d1cf421c6049dbbcf226700b89f7ae [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:46:50.120184 4755 raft_consensus.cc:3037] T 322d99ef5dd84b1e943963dc83c0a65e P d4111b368ec74fc6a14eeb9d6c9a6b89 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:46:50.120127 4688 leader_election.cc:302] T 322d99ef5dd84b1e943963dc83c0a65e P d4111b368ec74fc6a14eeb9d6c9a6b89 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4111b368ec74fc6a14eeb9d6c9a6b89, d9d1cf421c6049dbbcf226700b89f7ae; no voters:
I20260619 14:46:50.121484 4888 raft_consensus.cc:2454] T 322d99ef5dd84b1e943963dc83c0a65e P d9d1cf421c6049dbbcf226700b89f7ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e85f623b78bd484489fc27f261be42ed in term 1.
I20260619 14:46:50.121484 4755 raft_consensus.cc:2454] T 322d99ef5dd84b1e943963dc83c0a65e P d4111b368ec74fc6a14eeb9d6c9a6b89 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e85f623b78bd484489fc27f261be42ed in term 1.
I20260619 14:46:50.121635 4959 raft_consensus.cc:2741] T 322d99ef5dd84b1e943963dc83c0a65e P d4111b368ec74fc6a14eeb9d6c9a6b89 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260619 14:46:50.121793 4555 leader_election.cc:302] T 322d99ef5dd84b1e943963dc83c0a65e P e85f623b78bd484489fc27f261be42ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d4111b368ec74fc6a14eeb9d6c9a6b89, e85f623b78bd484489fc27f261be42ed; no voters:
I20260619 14:46:50.121978 4957 raft_consensus.cc:2781] T 322d99ef5dd84b1e943963dc83c0a65e P e85f623b78bd484489fc27f261be42ed [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:46:50.122290 4957 raft_consensus.cc:686] T 322d99ef5dd84b1e943963dc83c0a65e P e85f623b78bd484489fc27f261be42ed [term 1 LEADER]: Becoming Leader. State: Replica: e85f623b78bd484489fc27f261be42ed, State: Running, Role: LEADER
I20260619 14:46:50.122690 4957 consensus_queue.cc:237] T 322d99ef5dd84b1e943963dc83c0a65e P e85f623b78bd484489fc27f261be42ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d1cf421c6049dbbcf226700b89f7ae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34851 } } peers { permanent_uuid: "d4111b368ec74fc6a14eeb9d6c9a6b89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43135 } } peers { permanent_uuid: "e85f623b78bd484489fc27f261be42ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34845 } }
I20260619 14:46:50.126610 4478 catalog_manager.cc:5495] T 322d99ef5dd84b1e943963dc83c0a65e P e85f623b78bd484489fc27f261be42ed reported cstate change: term changed from 0 to 1, leader changed from <none> to e85f623b78bd484489fc27f261be42ed (127.0.98.1). New cstate: current_term: 1 leader_uuid: "e85f623b78bd484489fc27f261be42ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d1cf421c6049dbbcf226700b89f7ae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4111b368ec74fc6a14eeb9d6c9a6b89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43135 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e85f623b78bd484489fc27f261be42ed" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34845 } health_report { overall_health: HEALTHY } } }
I20260619 14:46:50.149243 4950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:50.145353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52804 (local address 127.0.98.1:34845)
0619 14:46:50.145486 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:50.145491 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:50.145609 (+ 118us) server_negotiation.cc:407] Connection header received
0619 14:46:50.145776 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:50.145799 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:50.145870 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:50.145991 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:46:50.146255 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:50.146787 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:50.147654 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:50.147894 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:50.148009 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:50.148333 (+ 324us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:46:50.148856 (+ 523us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:46:50.148995 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:50.149035 (+ 40us) server_negotiation.cc:299] Negotiation successful
0619 14:46:50.149094 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 14:46:50.156107 4755 raft_consensus.cc:1261] T 322d99ef5dd84b1e943963dc83c0a65e P d4111b368ec74fc6a14eeb9d6c9a6b89 [term 1 FOLLOWER]: Refusing update from remote peer e85f623b78bd484489fc27f261be42ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:46:50.156163 4888 raft_consensus.cc:1261] T 322d99ef5dd84b1e943963dc83c0a65e P d9d1cf421c6049dbbcf226700b89f7ae [term 1 FOLLOWER]: Refusing update from remote peer e85f623b78bd484489fc27f261be42ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:46:50.157188 4965 consensus_queue.cc:1035] T 322d99ef5dd84b1e943963dc83c0a65e P e85f623b78bd484489fc27f261be42ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4111b368ec74fc6a14eeb9d6c9a6b89" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:46:50.157563 4957 consensus_queue.cc:1035] T 322d99ef5dd84b1e943963dc83c0a65e P e85f623b78bd484489fc27f261be42ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9d1cf421c6049dbbcf226700b89f7ae" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:46:50.165406 4968 mvcc.cc:205] Tried to move back new op lower bound from 7298582159989530624 to 7298582159865049088. Current Snapshot: MvccSnapshot[applied={T|T < 7298582159989530624}]
I20260619 14:46:50.166507 4970 mvcc.cc:205] Tried to move back new op lower bound from 7298582159989530624 to 7298582159865049088. Current Snapshot: MvccSnapshot[applied={T|T < 7298582159989530624}]
I20260619 14:46:50.166754 4969 mvcc.cc:205] Tried to move back new op lower bound from 7298582159989530624 to 7298582159865049088. Current Snapshot: MvccSnapshot[applied={T|T < 7298582159989530624}]
I20260619 14:46:50.178503 4478 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38470:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:46:50.178727 4478 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38470:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:46:50.181603 4478 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ba356c35c2cb4fd38fc9830b168cc10a: Sending DeleteTablet for 3 replicas of tablet 322d99ef5dd84b1e943963dc83c0a65e
I20260619 14:46:50.182654 4735 tablet_service.cc:1514] Processing DeleteTablet for tablet 322d99ef5dd84b1e943963dc83c0a65e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:46:50 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37592
I20260619 14:46:50.182729 4602 tablet_service.cc:1514] Processing DeleteTablet for tablet 322d99ef5dd84b1e943963dc83c0a65e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:46:50 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52792
I20260619 14:46:50.182802 4868 tablet_service.cc:1514] Processing DeleteTablet for tablet 322d99ef5dd84b1e943963dc83c0a65e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:46:50 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38806
I20260619 14:46:50.183346 4979 tablet_replica.cc:330] T 322d99ef5dd84b1e943963dc83c0a65e P d9d1cf421c6049dbbcf226700b89f7ae: stopping tablet replica
I20260619 14:46:50.183529 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 4540
I20260619 14:46:50.183524 4978 tablet_replica.cc:330] T 322d99ef5dd84b1e943963dc83c0a65e P d4111b368ec74fc6a14eeb9d6c9a6b89: stopping tablet replica
I20260619 14:46:50.183804 4979 raft_consensus.cc:2229] T 322d99ef5dd84b1e943963dc83c0a65e P d9d1cf421c6049dbbcf226700b89f7ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:46:50.183856 4978 raft_consensus.cc:2229] T 322d99ef5dd84b1e943963dc83c0a65e P d4111b368ec74fc6a14eeb9d6c9a6b89 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:46:50.184239 4978 raft_consensus.cc:2258] T 322d99ef5dd84b1e943963dc83c0a65e P d4111b368ec74fc6a14eeb9d6c9a6b89 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:46:50.184252 4979 raft_consensus.cc:2258] T 322d99ef5dd84b1e943963dc83c0a65e P d9d1cf421c6049dbbcf226700b89f7ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:46:50.187124 4978 ts_tablet_manager.cc:1905] T 322d99ef5dd84b1e943963dc83c0a65e P d4111b368ec74fc6a14eeb9d6c9a6b89: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:46:50.187142 4979 ts_tablet_manager.cc:1905] T 322d99ef5dd84b1e943963dc83c0a65e P d9d1cf421c6049dbbcf226700b89f7ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:46:50.190397 4978 ts_tablet_manager.cc:1918] T 322d99ef5dd84b1e943963dc83c0a65e P d4111b368ec74fc6a14eeb9d6c9a6b89: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:46:50.190707 4978 log.cc:1196] T 322d99ef5dd84b1e943963dc83c0a65e P d4111b368ec74fc6a14eeb9d6c9a6b89: Deleting WAL directory at /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880383132139-392-0/minicluster-data/ts-1/wal/wals/322d99ef5dd84b1e943963dc83c0a65e
I20260619 14:46:50.190951 4979 ts_tablet_manager.cc:1918] T 322d99ef5dd84b1e943963dc83c0a65e P d9d1cf421c6049dbbcf226700b89f7ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:46:50.191069 4979 log.cc:1196] T 322d99ef5dd84b1e943963dc83c0a65e P d9d1cf421c6049dbbcf226700b89f7ae: Deleting WAL directory at /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781880383132139-392-0/minicluster-data/ts-2/wal/wals/322d99ef5dd84b1e943963dc83c0a65e
I20260619 14:46:50.191802 4979 ts_tablet_manager.cc:1939] T 322d99ef5dd84b1e943963dc83c0a65e P d9d1cf421c6049dbbcf226700b89f7ae: Deleting consensus metadata
I20260619 14:46:50.192009 4978 ts_tablet_manager.cc:1939] T 322d99ef5dd84b1e943963dc83c0a65e P d4111b368ec74fc6a14eeb9d6c9a6b89: Deleting consensus metadata
W20260619 14:46:50.192293 4465 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34845 (error 108)
I20260619 14:46:50.193485 4947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:50.193117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34845 (local address 127.0.0.1:52806)
0619 14:46:50.193272 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:50.193362 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260619 14:46:50.193446 4462 catalog_manager.cc:4841] TS d4111b368ec74fc6a14eeb9d6c9a6b89 (127.0.98.2:43135): tablet 322d99ef5dd84b1e943963dc83c0a65e (table test-table [id=e6d98526aeb0474fa6b5d089bcfadbd8]) successfully deleted
I20260619 14:46:50.193609 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 4673
W20260619 14:46:50.193746 4465 catalog_manager.cc:4584] TS e85f623b78bd484489fc27f261be42ed (127.0.98.1:34845): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 322d99ef5dd84b1e943963dc83c0a65e: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34845: connect: Connection refused (error 111)
I20260619 14:46:50.193754 4462 catalog_manager.cc:4841] TS d9d1cf421c6049dbbcf226700b89f7ae (127.0.98.3:34851): tablet 322d99ef5dd84b1e943963dc83c0a65e (table test-table [id=e6d98526aeb0474fa6b5d089bcfadbd8]) successfully deleted
I20260619 14:46:50.199805 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 4806
I20260619 14:46:50.206563 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 4447
2026-06-19T14:46:50Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2842 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-19T14:46:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:46:50Z Disabled control of system clock
I20260619 14:46:50.239012 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880383132139-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880383132139-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880383132139-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43005
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:39337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43005
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 14:46:50.360213 4984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:46:50.360566 4984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:46:50.360606 4984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:46:50.365052 4984 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:46:50.365217 4984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:46:50.365247 4984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:46:50.365280 4984 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:46:50.365339 4984 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:46:50.370905 4984 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880383132139-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880383132139-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43005
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43005
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4984
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880383132139-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:46:50.372376 4984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:46:50.373987 4984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 14:46:50.383805 4984 server_base.cc:1034] running on GCE node
W20260619 14:46:50.383731 4989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:50.383730 4992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:50.383731 4990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:50.385170 4984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:46:50.386579 4984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:46:50.387828 4984 hybrid_clock.cc:648] HybridClock initialized: now 1781880410387831 us; error 113 us; skew 500 ppm
I20260619 14:46:50.390429 4984 webserver.cc:460] Webserver started at http://127.0.98.62:39531/ using document root <none> and password file <none>
I20260619 14:46:50.391073 4984 fs_manager.cc:359] Metadata directory not provided
I20260619 14:46:50.391177 4984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:46:50.391423 4984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:46:50.393806 4984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880383132139-392-0/minicluster-data/master-0/data/instance:
uuid: "bf97387648be4ddcabb7f8859813082c"
format_stamp: "Formatted at 2026-06-19 14:46:50 on dist-test-slave-2sb1"
server_key: "82e067198e3644e6bf8188d71338be55"
server_key_iv: "08e650f3af921e2622aac46d7d8f01e9"
server_key_version: "encryptionkey@0"
I20260619 14:46:50.394434 4984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880383132139-392-0/minicluster-data/master-0/wal/instance:
uuid: "bf97387648be4ddcabb7f8859813082c"
format_stamp: "Formatted at 2026-06-19 14:46:50 on dist-test-slave-2sb1"
server_key: "82e067198e3644e6bf8188d71338be55"
server_key_iv: "08e650f3af921e2622aac46d7d8f01e9"
server_key_version: "encryptionkey@0"
I20260619 14:46:50.399003 4984 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 14:46:50.402143 4998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:50.404057 4984 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:46:50.404258 4984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880383132139-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880383132139-392-0/minicluster-data/master-0/wal
uuid: "bf97387648be4ddcabb7f8859813082c"
format_stamp: "Formatted at 2026-06-19 14:46:50 on dist-test-slave-2sb1"
server_key: "82e067198e3644e6bf8188d71338be55"
server_key_iv: "08e650f3af921e2622aac46d7d8f01e9"
server_key_version: "encryptionkey@0"
I20260619 14:46:50.404428 4984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880383132139-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880383132139-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880383132139-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:46:50.409495 4984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:46:50.410221 4984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:46:50.410434 4984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:46:50.418875 4984 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43005
I20260619 14:46:50.419193 5050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43005 every 8 connection(s)
I20260619 14:46:50.420043 4984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:46:50.423920 5051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:46:50.426719 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 4984
I20260619 14:46:50.426863 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880383132139-392-0/minicluster-data/master-0/wal/instance
I20260619 14:46:50.427289 392 external_mini_cluster.cc:1428] Setting key a8ca4d33a41c6ecc95aba2fd3912947f
I20260619 14:46:50.431391 5051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf97387648be4ddcabb7f8859813082c: Bootstrap starting.
I20260619 14:46:50.433844 5054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:50.429035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36248 (local address 127.0.98.62:43005)
0619 14:46:50.429762 (+ 727us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:50.429779 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:50.429818 (+ 39us) server_negotiation.cc:407] Connection header received
0619 14:46:50.430496 (+ 678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:50.430516 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:50.430901 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:50.431226 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:50.432189 (+ 963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:50.432194 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:50.432207 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:46:50.432217 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:50.432273 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:46:50.432289 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:50.432409 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:50.432569 (+ 160us) server_negotiation.cc:299] Negotiation successful
0619 14:46:50.433038 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":156,"threads_started":1}
I20260619 14:46:50.434680 5051 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf97387648be4ddcabb7f8859813082c: Neither blocks nor log segments found. Creating new log.
I20260619 14:46:50.435518 5051 log.cc:824] T 00000000000000000000000000000000 P bf97387648be4ddcabb7f8859813082c: Log is configured to *not* fsync() on all Append() calls
I20260619 14:46:50.438301 5051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf97387648be4ddcabb7f8859813082c: No bootstrap required, opened a new log
I20260619 14:46:50.442060 5051 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf97387648be4ddcabb7f8859813082c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf97387648be4ddcabb7f8859813082c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43005 } }
I20260619 14:46:50.442397 5051 raft_consensus.cc:374] T 00000000000000000000000000000000 P bf97387648be4ddcabb7f8859813082c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:46:50.442464 5051 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf97387648be4ddcabb7f8859813082c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf97387648be4ddcabb7f8859813082c, State: Initialized, Role: FOLLOWER
I20260619 14:46:50.442987 5051 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf97387648be4ddcabb7f8859813082c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf97387648be4ddcabb7f8859813082c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43005 } }
I20260619 14:46:50.443121 5051 raft_consensus.cc:388] T 00000000000000000000000000000000 P bf97387648be4ddcabb7f8859813082c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:46:50.443205 5051 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf97387648be4ddcabb7f8859813082c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:46:50.443303 5051 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf97387648be4ddcabb7f8859813082c [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:46:50.444311 5051 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf97387648be4ddcabb7f8859813082c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf97387648be4ddcabb7f8859813082c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43005 } }
I20260619 14:46:50.444738 5051 leader_election.cc:302] T 00000000000000000000000000000000 P bf97387648be4ddcabb7f8859813082c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf97387648be4ddcabb7f8859813082c; no voters:
I20260619 14:46:50.445096 5051 leader_election.cc:288] T 00000000000000000000000000000000 P bf97387648be4ddcabb7f8859813082c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:46:50.445225 5056 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf97387648be4ddcabb7f8859813082c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:46:50.445549 5056 raft_consensus.cc:686] T 00000000000000000000000000000000 P bf97387648be4ddcabb7f8859813082c [term 1 LEADER]: Becoming Leader. State: Replica: bf97387648be4ddcabb7f8859813082c, State: Running, Role: LEADER
I20260619 14:46:50.445888 5056 consensus_queue.cc:237] T 00000000000000000000000000000000 P bf97387648be4ddcabb7f8859813082c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf97387648be4ddcabb7f8859813082c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43005 } }
I20260619 14:46:50.446493 5051 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf97387648be4ddcabb7f8859813082c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:46:50.447084 5056 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf97387648be4ddcabb7f8859813082c [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf97387648be4ddcabb7f8859813082c. Latest consensus state: current_term: 1 leader_uuid: "bf97387648be4ddcabb7f8859813082c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf97387648be4ddcabb7f8859813082c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43005 } } }
I20260619 14:46:50.447207 5056 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf97387648be4ddcabb7f8859813082c [sys.catalog]: This master's current role is: LEADER
I20260619 14:46:50.447682 5057 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf97387648be4ddcabb7f8859813082c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bf97387648be4ddcabb7f8859813082c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf97387648be4ddcabb7f8859813082c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43005 } } }
I20260619 14:46:50.447794 5057 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf97387648be4ddcabb7f8859813082c [sys.catalog]: This master's current role is: LEADER
I20260619 14:46:50.447669 5062 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:46:50.451241 5062 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:46:50.458493 5062 catalog_manager.cc:1347] Generated new cluster ID: 067324b485ea4df7b5e3a004879840c5
I20260619 14:46:50.458592 5062 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:46:50.481657 5062 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:46:50.482060 5062 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:46:50.494490 5062 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bf97387648be4ddcabb7f8859813082c: Generated new TSK 0
I20260619 14:46:50.495434 5062 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:46:50.505419 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880383132139-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880383132139-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880383132139-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880383132139-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43005
--builtin_ntp_servers=127.0.98.20:39337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260619 14:46:50.637450 5075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:46:50.637773 5075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:46:50.637815 5075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:46:50.642232 5075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:46:50.642302 5075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:46:50.642364 5075 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 14:46:50.647996 5075 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880383132139-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880383132139-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880383132139-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43005
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5075
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880383132139-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:46:50.649196 5075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:46:50.650574 5075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:46:50.661088 5083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:50.661126 5080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:50.661095 5081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:50.662340 5075 server_base.cc:1034] running on GCE node
I20260619 14:46:50.662955 5075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:46:50.663705 5075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:46:50.664963 5075 hybrid_clock.cc:648] HybridClock initialized: now 1781880410664906 us; error 59 us; skew 500 ppm
I20260619 14:46:50.667397 5075 webserver.cc:460] Webserver started at http://127.0.98.1:44899/ using document root <none> and password file <none>
I20260619 14:46:50.668062 5075 fs_manager.cc:359] Metadata directory not provided
I20260619 14:46:50.668190 5075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:46:50.668484 5075 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:46:50.670818 5075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880383132139-392-0/minicluster-data/ts-0/data/instance:
uuid: "862fede6f5f1423a9de739dc7c93c28c"
format_stamp: "Formatted at 2026-06-19 14:46:50 on dist-test-slave-2sb1"
server_key: "d8cc26d9f45ab0f9ba76cac7e585fcea"
server_key_iv: "d858608f595ed8c32acecda190c47a26"
server_key_version: "encryptionkey@0"
I20260619 14:46:50.671504 5075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880383132139-392-0/minicluster-data/ts-0/wal/instance:
uuid: "862fede6f5f1423a9de739dc7c93c28c"
format_stamp: "Formatted at 2026-06-19 14:46:50 on dist-test-slave-2sb1"
server_key: "d8cc26d9f45ab0f9ba76cac7e585fcea"
server_key_iv: "d858608f595ed8c32acecda190c47a26"
server_key_version: "encryptionkey@0"
I20260619 14:46:50.675853 5075 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260619 14:46:50.679359 5089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:50.681160 5075 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:46:50.681345 5075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880383132139-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880383132139-392-0/minicluster-data/ts-0/wal
uuid: "862fede6f5f1423a9de739dc7c93c28c"
format_stamp: "Formatted at 2026-06-19 14:46:50 on dist-test-slave-2sb1"
server_key: "d8cc26d9f45ab0f9ba76cac7e585fcea"
server_key_iv: "d858608f595ed8c32acecda190c47a26"
server_key_version: "encryptionkey@0"
I20260619 14:46:50.681507 5075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880383132139-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880383132139-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880383132139-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:46:50.687283 5075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:46:50.688056 5075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:46:50.688282 5075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:46:50.689080 5075 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:46:50.690155 5075 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:46:50.690238 5075 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:50.690321 5075 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:46:50.690378 5075 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:50.702207 5075 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34839
I20260619 14:46:50.702225 5202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34839 every 8 connection(s)
I20260619 14:46:50.703366 5075 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880383132139-392-0/minicluster-data/ts-0/data/info.pb
I20260619 14:46:50.710835 5054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:50.705748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41375 (local address 127.0.98.62:43005)
0619 14:46:50.705940 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:50.705944 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:50.707166 (+ 1222us) server_negotiation.cc:407] Connection header received
0619 14:46:50.708817 (+ 1651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:50.708820 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:50.708873 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:50.708995 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:50.710281 (+ 1286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:50.710285 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:50.710288 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:46:50.710302 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:50.710333 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:46:50.710343 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:50.710450 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:50.710595 (+ 145us) server_negotiation.cc:299] Negotiation successful
0619 14:46:50.710673 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260619 14:46:50.711850 5205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:50.706063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43005 (local address 127.0.98.1:41375)
0619 14:46:50.706973 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:50.707022 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:50.708517 (+ 1495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:50.709188 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:50.709196 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:50.709297 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:46:50.709993 (+ 696us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:46:50.709994 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:46:50.710038 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:46:50.710114 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:46:50.710430 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:46:50.710438 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:46:50.710445 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 14:46:50.710667 (+ 222us) client_negotiation.cc:240] Negotiation successful
0619 14:46:50.710950 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":135,"threads_started":1}
I20260619 14:46:50.713531 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 5075
I20260619 14:46:50.713680 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880383132139-392-0/minicluster-data/ts-0/wal/instance
I20260619 14:46:50.713704 5203 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43005
I20260619 14:46:50.714052 392 external_mini_cluster.cc:1428] Setting key f2e60cf3de709ad3905ce0edcfafd6c0
I20260619 14:46:50.714128 5203 heartbeater.cc:464] Registering TS with master...
I20260619 14:46:50.714681 5203 heartbeater.cc:511] Master 127.0.98.62:43005 requested a full tablet report, sending...
I20260619 14:46:50.716698 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880383132139-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880383132139-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880383132139-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880383132139-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43005
--builtin_ntp_servers=127.0.98.20:39337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260619 14:46:50.716933 5015 ts_manager.cc:194] Registered new tserver with Master: 862fede6f5f1423a9de739dc7c93c28c (127.0.98.1:34839)
W20260619 14:46:50.842257 5206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:46:50.842598 5206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:46:50.842646 5206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:46:50.847760 5206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:46:50.847868 5206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:46:50.847962 5206 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 14:46:50.853740 5206 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880383132139-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880383132139-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880383132139-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43005
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5206
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880383132139-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:46:50.854964 5206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:46:50.856514 5206 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:46:50.868575 5212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:50.868726 5214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:50.868757 5211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:50.868916 5206 server_base.cc:1034] running on GCE node
I20260619 14:46:50.869794 5206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:46:50.870621 5206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:46:50.874141 5206 hybrid_clock.cc:648] HybridClock initialized: now 1781880410873972 us; error 186 us; skew 500 ppm
I20260619 14:46:50.877363 5206 webserver.cc:460] Webserver started at http://127.0.98.2:40859/ using document root <none> and password file <none>
I20260619 14:46:50.878167 5206 fs_manager.cc:359] Metadata directory not provided
I20260619 14:46:50.878261 5206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:46:50.878525 5206 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:46:50.881145 5206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880383132139-392-0/minicluster-data/ts-1/data/instance:
uuid: "e7c133f1c2a64b8d99f383b68be12333"
format_stamp: "Formatted at 2026-06-19 14:46:50 on dist-test-slave-2sb1"
server_key: "2a4234510fb10275028c8cf2ad0904ad"
server_key_iv: "8c688e1080cf2c5cdab60ba1a8a97684"
server_key_version: "encryptionkey@0"
I20260619 14:46:50.881863 5206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880383132139-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e7c133f1c2a64b8d99f383b68be12333"
format_stamp: "Formatted at 2026-06-19 14:46:50 on dist-test-slave-2sb1"
server_key: "2a4234510fb10275028c8cf2ad0904ad"
server_key_iv: "8c688e1080cf2c5cdab60ba1a8a97684"
server_key_version: "encryptionkey@0"
I20260619 14:46:50.886960 5206 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 14:46:50.890389 5220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:50.892450 5206 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260619 14:46:50.892864 5206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880383132139-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880383132139-392-0/minicluster-data/ts-1/wal
uuid: "e7c133f1c2a64b8d99f383b68be12333"
format_stamp: "Formatted at 2026-06-19 14:46:50 on dist-test-slave-2sb1"
server_key: "2a4234510fb10275028c8cf2ad0904ad"
server_key_iv: "8c688e1080cf2c5cdab60ba1a8a97684"
server_key_version: "encryptionkey@0"
I20260619 14:46:50.893040 5206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880383132139-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880383132139-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880383132139-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:46:50.898510 5206 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:46:50.899214 5206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:46:50.899441 5206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:46:50.900066 5206 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:46:50.901662 5206 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:46:50.901927 5206 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:50.902112 5206 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:46:50.902217 5206 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:50.913930 5206 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41237
I20260619 14:46:50.913969 5333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41237 every 8 connection(s)
I20260619 14:46:50.914994 5206 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880383132139-392-0/minicluster-data/ts-1/data/info.pb
I20260619 14:46:50.915900 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 5206
I20260619 14:46:50.916038 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880383132139-392-0/minicluster-data/ts-1/wal/instance
I20260619 14:46:50.916433 392 external_mini_cluster.cc:1428] Setting key 00681e7b259b285f28a6a6d887232e87
I20260619 14:46:50.918982 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880383132139-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880383132139-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880383132139-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880383132139-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43005
--builtin_ntp_servers=127.0.98.20:39337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260619 14:46:50.923023 5054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:50.917389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42193 (local address 127.0.98.62:43005)
0619 14:46:50.917554 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:50.917559 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:50.919087 (+ 1528us) server_negotiation.cc:407] Connection header received
0619 14:46:50.920759 (+ 1672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:50.920763 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:50.920814 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:50.920907 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:50.922489 (+ 1582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:50.922493 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:50.922496 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:46:50.922511 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:50.922534 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:46:50.922543 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:50.922642 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:50.922825 (+ 183us) server_negotiation.cc:299] Negotiation successful
0619 14:46:50.922870 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 14:46:50.924487 5336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:50.917996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43005 (local address 127.0.98.2:42193)
0619 14:46:50.918897 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:50.918952 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:50.920459 (+ 1507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:50.921101 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:50.921109 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:50.921213 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:46:50.921943 (+ 730us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:46:50.921947 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:46:50.922058 (+ 111us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:46:50.922161 (+ 103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:46:50.922653 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:46:50.922666 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:46:50.922675 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 14:46:50.922872 (+ 197us) client_negotiation.cc:240] Negotiation successful
0619 14:46:50.923281 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":195,"threads_started":1}
I20260619 14:46:50.926267 5334 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43005
I20260619 14:46:50.926570 5334 heartbeater.cc:464] Registering TS with master...
I20260619 14:46:50.927074 5334 heartbeater.cc:511] Master 127.0.98.62:43005 requested a full tablet report, sending...
I20260619 14:46:50.928368 5015 ts_manager.cc:194] Registered new tserver with Master: e7c133f1c2a64b8d99f383b68be12333 (127.0.98.2:41237)
W20260619 14:46:51.046597 5337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:46:51.046981 5337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:46:51.047031 5337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:46:51.052773 5337 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:46:51.052881 5337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:46:51.053035 5337 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 14:46:51.061990 5337 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880383132139-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880383132139-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880383132139-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43005
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5337
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880383132139-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:46:51.063607 5337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:46:51.065589 5337 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:46:51.077623 5345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:51.077636 5342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:51.077623 5343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:51.078281 5337 server_base.cc:1034] running on GCE node
I20260619 14:46:51.078788 5337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:46:51.079561 5337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:46:51.080781 5337 hybrid_clock.cc:648] HybridClock initialized: now 1781880411080755 us; error 46 us; skew 500 ppm
I20260619 14:46:51.084450 5337 webserver.cc:460] Webserver started at http://127.0.98.3:45077/ using document root <none> and password file <none>
I20260619 14:46:51.085242 5337 fs_manager.cc:359] Metadata directory not provided
I20260619 14:46:51.085386 5337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:46:51.085799 5337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:46:51.088388 5337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880383132139-392-0/minicluster-data/ts-2/data/instance:
uuid: "9565ae1d8e994fb3a74b4ecec30e6f61"
format_stamp: "Formatted at 2026-06-19 14:46:51 on dist-test-slave-2sb1"
server_key: "965603b72f6b1df86d89adbbd90760d2"
server_key_iv: "48c269fb75c4fdecafe72606c10985de"
server_key_version: "encryptionkey@0"
I20260619 14:46:51.089169 5337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880383132139-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9565ae1d8e994fb3a74b4ecec30e6f61"
format_stamp: "Formatted at 2026-06-19 14:46:51 on dist-test-slave-2sb1"
server_key: "965603b72f6b1df86d89adbbd90760d2"
server_key_iv: "48c269fb75c4fdecafe72606c10985de"
server_key_version: "encryptionkey@0"
I20260619 14:46:51.094607 5337 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 14:46:51.098404 5351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:51.100102 5337 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 14:46:51.100440 5337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880383132139-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880383132139-392-0/minicluster-data/ts-2/wal
uuid: "9565ae1d8e994fb3a74b4ecec30e6f61"
format_stamp: "Formatted at 2026-06-19 14:46:51 on dist-test-slave-2sb1"
server_key: "965603b72f6b1df86d89adbbd90760d2"
server_key_iv: "48c269fb75c4fdecafe72606c10985de"
server_key_version: "encryptionkey@0"
I20260619 14:46:51.100594 5337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880383132139-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880383132139-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880383132139-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:46:51.106781 5337 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:46:51.107735 5337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:46:51.108103 5337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:46:51.109027 5337 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:46:51.110052 5337 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:46:51.110103 5337 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:51.110147 5337 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:46:51.110203 5337 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:51.120884 5337 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35187
I20260619 14:46:51.120922 5464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35187 every 8 connection(s)
I20260619 14:46:51.121938 5337 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880383132139-392-0/minicluster-data/ts-2/data/info.pb
I20260619 14:46:51.127703 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 5337
I20260619 14:46:51.127827 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880383132139-392-0/minicluster-data/ts-2/wal/instance
I20260619 14:46:51.128348 392 external_mini_cluster.cc:1428] Setting key bc7c299d054137d247a38791f32d4af8
I20260619 14:46:51.129604 5054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:51.124095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55257 (local address 127.0.98.62:43005)
0619 14:46:51.124263 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:51.124266 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:51.125772 (+ 1506us) server_negotiation.cc:407] Connection header received
0619 14:46:51.127340 (+ 1568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:51.127344 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:51.127399 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:51.127525 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:51.129104 (+ 1579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:51.129107 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:51.129110 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:46:51.129124 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:51.129146 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:46:51.129153 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:51.129261 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:51.129420 (+ 159us) server_negotiation.cc:299] Negotiation successful
0619 14:46:51.129468 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 14:46:51.130810 5467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:51.124721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43005 (local address 127.0.98.3:55257)
0619 14:46:51.125564 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:51.125613 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:51.127043 (+ 1430us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:51.127732 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:51.127741 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:51.127919 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:46:51.128761 (+ 842us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:46:51.128763 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:46:51.128814 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:46:51.128904 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:46:51.129249 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:46:51.129257 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:46:51.129263 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 14:46:51.129462 (+ 199us) client_negotiation.cc:240] Negotiation successful
0619 14:46:51.129743 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":143,"threads_started":1}
I20260619 14:46:51.132056 5465 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43005
I20260619 14:46:51.132344 5465 heartbeater.cc:464] Registering TS with master...
I20260619 14:46:51.132988 5465 heartbeater.cc:511] Master 127.0.98.62:43005 requested a full tablet report, sending...
I20260619 14:46:51.134191 5012 ts_manager.cc:194] Registered new tserver with Master: 9565ae1d8e994fb3a74b4ecec30e6f61 (127.0.98.3:35187)
I20260619 14:46:51.144563 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:46:51.149951 5054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:51.147657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36256 (local address 127.0.98.62:43005)
0619 14:46:51.147805 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:51.147809 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:51.148094 (+ 285us) server_negotiation.cc:407] Connection header received
0619 14:46:51.148152 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:51.148156 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:51.148208 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:51.148316 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:51.149304 (+ 988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:51.149308 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:51.149312 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:46:51.149473 (+ 161us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:51.149496 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:46:51.149507 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:51.149637 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:51.149725 (+ 88us) server_negotiation.cc:299] Negotiation successful
0619 14:46:51.149789 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 14:46:51.155189 5012 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36256:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:46:51.158398 5012 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:46:51.175720 5476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:51.171464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41237 (local address 127.0.0.1:43382)
0619 14:46:51.172179 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:51.172238 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:51.172380 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:51.173942 (+ 1562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:51.173945 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:51.173962 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:46:51.174723 (+ 761us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:46:51.174730 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:46:51.174758 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:46:51.174839 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:46:51.175316 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:46:51.175326 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:46:51.175333 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 14:46:51.175445 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 14:46:51.175527 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":107,"threads_started":1}
I20260619 14:46:51.175948 5480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:51.172523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56774 (local address 127.0.98.3:35187)
0619 14:46:51.172866 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:51.172889 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:51.173251 (+ 362us) server_negotiation.cc:407] Connection header received
0619 14:46:51.173321 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:51.173330 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:51.173578 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:51.173747 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:51.175161 (+ 1414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:51.175169 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:51.175184 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:46:51.175199 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:51.175251 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:46:51.175288 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:51.175424 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:51.175638 (+ 214us) server_negotiation.cc:299] Negotiation successful
0619 14:46:51.175722 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":115,"threads_started":1}
I20260619 14:46:51.175959 5478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:51.172258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34839 (local address 127.0.0.1:37064)
0619 14:46:51.173360 (+ 1102us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:51.173383 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:51.173496 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:51.174781 (+ 1285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:51.174784 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:51.174800 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:46:51.175307 (+ 507us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:46:51.175312 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:46:51.175325 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:46:51.175357 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:46:51.175693 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:46:51.175696 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:46:51.175700 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 14:46:51.175778 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 14:46:51.175815 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":881,"thread_start_us":99,"threads_started":1}
I20260619 14:46:51.176318 5479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:51.172062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35187 (local address 127.0.0.1:56774)
0619 14:46:51.173012 (+ 950us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:51.173029 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:51.173136 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:51.173764 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:51.173771 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:51.173858 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:46:51.174684 (+ 826us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:46:51.174969 (+ 285us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:46:51.174988 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:46:51.175015 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:46:51.175471 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:46:51.175474 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:46:51.175478 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 14:46:51.176115 (+ 637us) client_negotiation.cc:240] Negotiation successful
0619 14:46:51.176159 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":864,"thread_start_us":94,"threads_started":1}
I20260619 14:46:51.176986 5481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:51.172805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37064 (local address 127.0.98.1:34839)
0619 14:46:51.173292 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:51.173305 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:51.173605 (+ 300us) server_negotiation.cc:407] Connection header received
0619 14:46:51.173704 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:51.173709 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:51.173982 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:51.174723 (+ 741us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:51.175458 (+ 735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:51.175463 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:51.175474 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:46:51.175486 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:51.175522 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:46:51.175548 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:51.175650 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:51.176723 (+ 1073us) server_negotiation.cc:299] Negotiation successful
0619 14:46:51.176806 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":165,"threads_started":1}
I20260619 14:46:51.177557 5477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:51.171940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43382 (local address 127.0.98.2:41237)
0619 14:46:51.173509 (+ 1569us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:51.173520 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:51.173544 (+ 24us) server_negotiation.cc:407] Connection header received
0619 14:46:51.173595 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:51.173600 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:51.173771 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:51.174223 (+ 452us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:51.175016 (+ 793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:51.175022 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:51.175039 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:46:51.175056 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:51.175108 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:46:51.175167 (+ 59us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:51.176450 (+ 1283us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:51.176512 (+ 62us) server_negotiation.cc:299] Negotiation successful
0619 14:46:51.177365 (+ 853us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1467,"thread_start_us":167,"threads_started":1}
I20260619 14:46:51.179955 5399 tablet_service.cc:1467] Processing CreateTablet for tablet aabf23754c4e4d8f9055333d73e7d7d1 (DEFAULT_TABLE table=test-table [id=dd6732e93a1c4129b979871672b24846]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:46:51.179940 5137 tablet_service.cc:1467] Processing CreateTablet for tablet aabf23754c4e4d8f9055333d73e7d7d1 (DEFAULT_TABLE table=test-table [id=dd6732e93a1c4129b979871672b24846]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:46:51.180537 5268 tablet_service.cc:1467] Processing CreateTablet for tablet aabf23754c4e4d8f9055333d73e7d7d1 (DEFAULT_TABLE table=test-table [id=dd6732e93a1c4129b979871672b24846]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:46:51.181262 5137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aabf23754c4e4d8f9055333d73e7d7d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:46:51.181262 5399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aabf23754c4e4d8f9055333d73e7d7d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:46:51.181561 5268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aabf23754c4e4d8f9055333d73e7d7d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:46:51.191025 5484 tablet_bootstrap.cc:492] T aabf23754c4e4d8f9055333d73e7d7d1 P e7c133f1c2a64b8d99f383b68be12333: Bootstrap starting.
I20260619 14:46:51.192049 5482 tablet_bootstrap.cc:492] T aabf23754c4e4d8f9055333d73e7d7d1 P 862fede6f5f1423a9de739dc7c93c28c: Bootstrap starting.
I20260619 14:46:51.193419 5483 tablet_bootstrap.cc:492] T aabf23754c4e4d8f9055333d73e7d7d1 P 9565ae1d8e994fb3a74b4ecec30e6f61: Bootstrap starting.
I20260619 14:46:51.193984 5484 tablet_bootstrap.cc:654] T aabf23754c4e4d8f9055333d73e7d7d1 P e7c133f1c2a64b8d99f383b68be12333: Neither blocks nor log segments found. Creating new log.
I20260619 14:46:51.194834 5484 log.cc:824] T aabf23754c4e4d8f9055333d73e7d7d1 P e7c133f1c2a64b8d99f383b68be12333: Log is configured to *not* fsync() on all Append() calls
I20260619 14:46:51.195456 5482 tablet_bootstrap.cc:654] T aabf23754c4e4d8f9055333d73e7d7d1 P 862fede6f5f1423a9de739dc7c93c28c: Neither blocks nor log segments found. Creating new log.
I20260619 14:46:51.196206 5483 tablet_bootstrap.cc:654] T aabf23754c4e4d8f9055333d73e7d7d1 P 9565ae1d8e994fb3a74b4ecec30e6f61: Neither blocks nor log segments found. Creating new log.
I20260619 14:46:51.196493 5482 log.cc:824] T aabf23754c4e4d8f9055333d73e7d7d1 P 862fede6f5f1423a9de739dc7c93c28c: Log is configured to *not* fsync() on all Append() calls
I20260619 14:46:51.197305 5484 tablet_bootstrap.cc:492] T aabf23754c4e4d8f9055333d73e7d7d1 P e7c133f1c2a64b8d99f383b68be12333: No bootstrap required, opened a new log
I20260619 14:46:51.197314 5483 log.cc:824] T aabf23754c4e4d8f9055333d73e7d7d1 P 9565ae1d8e994fb3a74b4ecec30e6f61: Log is configured to *not* fsync() on all Append() calls
I20260619 14:46:51.197535 5484 ts_tablet_manager.cc:1397] T aabf23754c4e4d8f9055333d73e7d7d1 P e7c133f1c2a64b8d99f383b68be12333: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260619 14:46:51.198936 5482 tablet_bootstrap.cc:492] T aabf23754c4e4d8f9055333d73e7d7d1 P 862fede6f5f1423a9de739dc7c93c28c: No bootstrap required, opened a new log
I20260619 14:46:51.199146 5482 ts_tablet_manager.cc:1397] T aabf23754c4e4d8f9055333d73e7d7d1 P 862fede6f5f1423a9de739dc7c93c28c: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260619 14:46:51.199476 5483 tablet_bootstrap.cc:492] T aabf23754c4e4d8f9055333d73e7d7d1 P 9565ae1d8e994fb3a74b4ecec30e6f61: No bootstrap required, opened a new log
I20260619 14:46:51.199661 5483 ts_tablet_manager.cc:1397] T aabf23754c4e4d8f9055333d73e7d7d1 P 9565ae1d8e994fb3a74b4ecec30e6f61: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260619 14:46:51.201741 5484 raft_consensus.cc:348] T aabf23754c4e4d8f9055333d73e7d7d1 P e7c133f1c2a64b8d99f383b68be12333 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c133f1c2a64b8d99f383b68be12333" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41237 } } peers { permanent_uuid: "862fede6f5f1423a9de739dc7c93c28c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34839 } } peers { permanent_uuid: "9565ae1d8e994fb3a74b4ecec30e6f61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35187 } }
I20260619 14:46:51.202018 5484 raft_consensus.cc:374] T aabf23754c4e4d8f9055333d73e7d7d1 P e7c133f1c2a64b8d99f383b68be12333 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:46:51.202092 5484 raft_consensus.cc:729] T aabf23754c4e4d8f9055333d73e7d7d1 P e7c133f1c2a64b8d99f383b68be12333 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7c133f1c2a64b8d99f383b68be12333, State: Initialized, Role: FOLLOWER
I20260619 14:46:51.202627 5484 consensus_queue.cc:260] T aabf23754c4e4d8f9055333d73e7d7d1 P e7c133f1c2a64b8d99f383b68be12333 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c133f1c2a64b8d99f383b68be12333" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41237 } } peers { permanent_uuid: "862fede6f5f1423a9de739dc7c93c28c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34839 } } peers { permanent_uuid: "9565ae1d8e994fb3a74b4ecec30e6f61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35187 } }
I20260619 14:46:51.203785 5484 ts_tablet_manager.cc:1428] T aabf23754c4e4d8f9055333d73e7d7d1 P e7c133f1c2a64b8d99f383b68be12333: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 14:46:51.204166 5334 heartbeater.cc:503] Master 127.0.98.62:43005 was elected leader, sending a full tablet report...
I20260619 14:46:51.204268 5483 raft_consensus.cc:348] T aabf23754c4e4d8f9055333d73e7d7d1 P 9565ae1d8e994fb3a74b4ecec30e6f61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c133f1c2a64b8d99f383b68be12333" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41237 } } peers { permanent_uuid: "862fede6f5f1423a9de739dc7c93c28c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34839 } } peers { permanent_uuid: "9565ae1d8e994fb3a74b4ecec30e6f61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35187 } }
I20260619 14:46:51.204563 5483 raft_consensus.cc:374] T aabf23754c4e4d8f9055333d73e7d7d1 P 9565ae1d8e994fb3a74b4ecec30e6f61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:46:51.204268 5482 raft_consensus.cc:348] T aabf23754c4e4d8f9055333d73e7d7d1 P 862fede6f5f1423a9de739dc7c93c28c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c133f1c2a64b8d99f383b68be12333" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41237 } } peers { permanent_uuid: "862fede6f5f1423a9de739dc7c93c28c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34839 } } peers { permanent_uuid: "9565ae1d8e994fb3a74b4ecec30e6f61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35187 } }
I20260619 14:46:51.204738 5482 raft_consensus.cc:374] T aabf23754c4e4d8f9055333d73e7d7d1 P 862fede6f5f1423a9de739dc7c93c28c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:46:51.204768 5483 raft_consensus.cc:729] T aabf23754c4e4d8f9055333d73e7d7d1 P 9565ae1d8e994fb3a74b4ecec30e6f61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9565ae1d8e994fb3a74b4ecec30e6f61, State: Initialized, Role: FOLLOWER
I20260619 14:46:51.204815 5482 raft_consensus.cc:729] T aabf23754c4e4d8f9055333d73e7d7d1 P 862fede6f5f1423a9de739dc7c93c28c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 862fede6f5f1423a9de739dc7c93c28c, State: Initialized, Role: FOLLOWER
I20260619 14:46:51.205513 5482 consensus_queue.cc:260] T aabf23754c4e4d8f9055333d73e7d7d1 P 862fede6f5f1423a9de739dc7c93c28c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c133f1c2a64b8d99f383b68be12333" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41237 } } peers { permanent_uuid: "862fede6f5f1423a9de739dc7c93c28c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34839 } } peers { permanent_uuid: "9565ae1d8e994fb3a74b4ecec30e6f61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35187 } }
I20260619 14:46:51.205508 5483 consensus_queue.cc:260] T aabf23754c4e4d8f9055333d73e7d7d1 P 9565ae1d8e994fb3a74b4ecec30e6f61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c133f1c2a64b8d99f383b68be12333" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41237 } } peers { permanent_uuid: "862fede6f5f1423a9de739dc7c93c28c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34839 } } peers { permanent_uuid: "9565ae1d8e994fb3a74b4ecec30e6f61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35187 } }
I20260619 14:46:51.206341 5203 heartbeater.cc:503] Master 127.0.98.62:43005 was elected leader, sending a full tablet report...
I20260619 14:46:51.206772 5483 ts_tablet_manager.cc:1428] T aabf23754c4e4d8f9055333d73e7d7d1 P 9565ae1d8e994fb3a74b4ecec30e6f61: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260619 14:46:51.207156 5465 heartbeater.cc:503] Master 127.0.98.62:43005 was elected leader, sending a full tablet report...
I20260619 14:46:51.208001 5482 ts_tablet_manager.cc:1428] T aabf23754c4e4d8f9055333d73e7d7d1 P 862fede6f5f1423a9de739dc7c93c28c: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
I20260619 14:46:51.233049 5490 raft_consensus.cc:482] T aabf23754c4e4d8f9055333d73e7d7d1 P 9565ae1d8e994fb3a74b4ecec30e6f61 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:46:51.233314 5490 raft_consensus.cc:504] T aabf23754c4e4d8f9055333d73e7d7d1 P 9565ae1d8e994fb3a74b4ecec30e6f61 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c133f1c2a64b8d99f383b68be12333" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41237 } } peers { permanent_uuid: "862fede6f5f1423a9de739dc7c93c28c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34839 } } peers { permanent_uuid: "9565ae1d8e994fb3a74b4ecec30e6f61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35187 } }
I20260619 14:46:51.234616 5490 leader_election.cc:288] T aabf23754c4e4d8f9055333d73e7d7d1 P 9565ae1d8e994fb3a74b4ecec30e6f61 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e7c133f1c2a64b8d99f383b68be12333 (127.0.98.2:41237), 862fede6f5f1423a9de739dc7c93c28c (127.0.98.1:34839)
I20260619 14:46:51.237453 5467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:51.235140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41237 (local address 127.0.98.3:36287)
0619 14:46:51.235352 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:51.235381 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:51.235486 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:51.235991 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:51.235995 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:51.236011 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:46:51.236612 (+ 601us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:46:51.236615 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:46:51.236628 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:46:51.236677 (+ 49us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:46:51.237140 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:46:51.237143 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:46:51.237146 (+ 3us) client_negotiation.cc:769] Sending connection context
0619 14:46:51.237265 (+ 119us) client_negotiation.cc:240] Negotiation successful
0619 14:46:51.237307 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119}
I20260619 14:46:51.237453 5477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:51.235327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36287 (local address 127.0.98.2:41237)
0619 14:46:51.235668 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:51.235673 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:51.235697 (+ 24us) server_negotiation.cc:407] Connection header received
0619 14:46:51.235741 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:51.235743 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:51.235775 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:51.235895 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:51.236799 (+ 904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:51.236805 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:51.236811 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:46:51.236828 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:51.236862 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:46:51.236878 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:51.237027 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:51.237266 (+ 239us) server_negotiation.cc:299] Negotiation successful
0619 14:46:51.237317 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 14:46:51.238447 5491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:51.235541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34839 (local address 127.0.98.3:33245)
0619 14:46:51.235876 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:51.235892 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:51.235968 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:51.236626 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:51.236629 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:51.236685 (+ 56us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:46:51.237665 (+ 980us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:46:51.237667 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:46:51.237679 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:46:51.237707 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:46:51.238172 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:46:51.238179 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:46:51.238184 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 14:46:51.238269 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 14:46:51.238312 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":129,"threads_started":1}
I20260619 14:46:51.238523 5288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aabf23754c4e4d8f9055333d73e7d7d1" candidate_uuid: "9565ae1d8e994fb3a74b4ecec30e6f61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7c133f1c2a64b8d99f383b68be12333" is_pre_election: true
I20260619 14:46:51.238631 5481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:51.235929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33245 (local address 127.0.98.1:34839)
0619 14:46:51.236131 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:51.236134 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:51.236146 (+ 12us) server_negotiation.cc:407] Connection header received
0619 14:46:51.236220 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:51.236223 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:51.236275 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:51.236362 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:51.238000 (+ 1638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:51.238004 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:51.238007 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:46:51.238022 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:51.238046 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:46:51.238062 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:51.238194 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:51.238269 (+ 75us) server_negotiation.cc:299] Negotiation successful
0619 14:46:51.238477 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 14:46:51.238883 5288 raft_consensus.cc:2454] T aabf23754c4e4d8f9055333d73e7d7d1 P e7c133f1c2a64b8d99f383b68be12333 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9565ae1d8e994fb3a74b4ecec30e6f61 in term 0.
I20260619 14:46:51.239396 5355 leader_election.cc:302] T aabf23754c4e4d8f9055333d73e7d7d1 P 9565ae1d8e994fb3a74b4ecec30e6f61 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9565ae1d8e994fb3a74b4ecec30e6f61, e7c133f1c2a64b8d99f383b68be12333; no voters:
I20260619 14:46:51.239737 5490 raft_consensus.cc:2781] T aabf23754c4e4d8f9055333d73e7d7d1 P 9565ae1d8e994fb3a74b4ecec30e6f61 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:46:51.239848 5490 raft_consensus.cc:482] T aabf23754c4e4d8f9055333d73e7d7d1 P 9565ae1d8e994fb3a74b4ecec30e6f61 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:46:51.239912 5490 raft_consensus.cc:3037] T aabf23754c4e4d8f9055333d73e7d7d1 P 9565ae1d8e994fb3a74b4ecec30e6f61 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:46:51.239843 5157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aabf23754c4e4d8f9055333d73e7d7d1" candidate_uuid: "9565ae1d8e994fb3a74b4ecec30e6f61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862fede6f5f1423a9de739dc7c93c28c" is_pre_election: true
I20260619 14:46:51.240163 5157 raft_consensus.cc:2454] T aabf23754c4e4d8f9055333d73e7d7d1 P 862fede6f5f1423a9de739dc7c93c28c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9565ae1d8e994fb3a74b4ecec30e6f61 in term 0.
I20260619 14:46:51.241277 5490 raft_consensus.cc:504] T aabf23754c4e4d8f9055333d73e7d7d1 P 9565ae1d8e994fb3a74b4ecec30e6f61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c133f1c2a64b8d99f383b68be12333" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41237 } } peers { permanent_uuid: "862fede6f5f1423a9de739dc7c93c28c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34839 } } peers { permanent_uuid: "9565ae1d8e994fb3a74b4ecec30e6f61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35187 } }
I20260619 14:46:51.241621 5490 leader_election.cc:288] T aabf23754c4e4d8f9055333d73e7d7d1 P 9565ae1d8e994fb3a74b4ecec30e6f61 [CANDIDATE]: Term 1 election: Requested vote from peers e7c133f1c2a64b8d99f383b68be12333 (127.0.98.2:41237), 862fede6f5f1423a9de739dc7c93c28c (127.0.98.1:34839)
I20260619 14:46:51.242103 5288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aabf23754c4e4d8f9055333d73e7d7d1" candidate_uuid: "9565ae1d8e994fb3a74b4ecec30e6f61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7c133f1c2a64b8d99f383b68be12333"
I20260619 14:46:51.242169 5157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aabf23754c4e4d8f9055333d73e7d7d1" candidate_uuid: "9565ae1d8e994fb3a74b4ecec30e6f61" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "862fede6f5f1423a9de739dc7c93c28c"
I20260619 14:46:51.242268 5288 raft_consensus.cc:3037] T aabf23754c4e4d8f9055333d73e7d7d1 P e7c133f1c2a64b8d99f383b68be12333 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:46:51.242340 5157 raft_consensus.cc:3037] T aabf23754c4e4d8f9055333d73e7d7d1 P 862fede6f5f1423a9de739dc7c93c28c [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:46:51.243520 5288 raft_consensus.cc:2454] T aabf23754c4e4d8f9055333d73e7d7d1 P e7c133f1c2a64b8d99f383b68be12333 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9565ae1d8e994fb3a74b4ecec30e6f61 in term 1.
I20260619 14:46:51.243523 5157 raft_consensus.cc:2454] T aabf23754c4e4d8f9055333d73e7d7d1 P 862fede6f5f1423a9de739dc7c93c28c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9565ae1d8e994fb3a74b4ecec30e6f61 in term 1.
I20260619 14:46:51.243884 5355 leader_election.cc:302] T aabf23754c4e4d8f9055333d73e7d7d1 P 9565ae1d8e994fb3a74b4ecec30e6f61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9565ae1d8e994fb3a74b4ecec30e6f61, e7c133f1c2a64b8d99f383b68be12333; no voters:
I20260619 14:46:51.244099 5490 raft_consensus.cc:2781] T aabf23754c4e4d8f9055333d73e7d7d1 P 9565ae1d8e994fb3a74b4ecec30e6f61 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:46:51.244352 5490 raft_consensus.cc:686] T aabf23754c4e4d8f9055333d73e7d7d1 P 9565ae1d8e994fb3a74b4ecec30e6f61 [term 1 LEADER]: Becoming Leader. State: Replica: 9565ae1d8e994fb3a74b4ecec30e6f61, State: Running, Role: LEADER
I20260619 14:46:51.244773 5490 consensus_queue.cc:237] T aabf23754c4e4d8f9055333d73e7d7d1 P 9565ae1d8e994fb3a74b4ecec30e6f61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c133f1c2a64b8d99f383b68be12333" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41237 } } peers { permanent_uuid: "862fede6f5f1423a9de739dc7c93c28c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34839 } } peers { permanent_uuid: "9565ae1d8e994fb3a74b4ecec30e6f61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35187 } }
I20260619 14:46:51.249423 5012 catalog_manager.cc:5495] T aabf23754c4e4d8f9055333d73e7d7d1 P 9565ae1d8e994fb3a74b4ecec30e6f61 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9565ae1d8e994fb3a74b4ecec30e6f61 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "9565ae1d8e994fb3a74b4ecec30e6f61" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7c133f1c2a64b8d99f383b68be12333" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "862fede6f5f1423a9de739dc7c93c28c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34839 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9565ae1d8e994fb3a74b4ecec30e6f61" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35187 } health_report { overall_health: HEALTHY } } }
I20260619 14:46:51.260350 5480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:51.258469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56784 (local address 127.0.98.3:35187)
0619 14:46:51.258635 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:51.258642 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:51.258726 (+ 84us) server_negotiation.cc:407] Connection header received
0619 14:46:51.258946 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:51.258949 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:51.259010 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:51.259107 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:51.259920 (+ 813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:51.259923 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:51.259926 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:46:51.259939 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:51.259960 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:46:51.259971 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:51.260088 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:51.260151 (+ 63us) server_negotiation.cc:299] Negotiation successful
0619 14:46:51.260194 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 14:46:51.267988 5157 raft_consensus.cc:1261] T aabf23754c4e4d8f9055333d73e7d7d1 P 862fede6f5f1423a9de739dc7c93c28c [term 1 FOLLOWER]: Refusing update from remote peer 9565ae1d8e994fb3a74b4ecec30e6f61: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:46:51.268020 5288 raft_consensus.cc:1261] T aabf23754c4e4d8f9055333d73e7d7d1 P e7c133f1c2a64b8d99f383b68be12333 [term 1 FOLLOWER]: Refusing update from remote peer 9565ae1d8e994fb3a74b4ecec30e6f61: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:46:51.268878 5490 consensus_queue.cc:1035] T aabf23754c4e4d8f9055333d73e7d7d1 P 9565ae1d8e994fb3a74b4ecec30e6f61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "862fede6f5f1423a9de739dc7c93c28c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:46:51.269164 5492 consensus_queue.cc:1035] T aabf23754c4e4d8f9055333d73e7d7d1 P 9565ae1d8e994fb3a74b4ecec30e6f61 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7c133f1c2a64b8d99f383b68be12333" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:46:51.278934 5497 mvcc.cc:205] Tried to move back new op lower bound from 7298582164544589824 to 7298582164462858240. Current Snapshot: MvccSnapshot[applied={T|T < 7298582164544589824}]
I20260619 14:46:51.288048 5012 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36256:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:46:51.288228 5012 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36256:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:46:51.291119 5012 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bf97387648be4ddcabb7f8859813082c: Sending DeleteTablet for 3 replicas of tablet aabf23754c4e4d8f9055333d73e7d7d1
I20260619 14:46:51.292344 5399 tablet_service.cc:1514] Processing DeleteTablet for tablet aabf23754c4e4d8f9055333d73e7d7d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:46:51 UTC) from {username='slave'} at 127.0.0.1:56774
I20260619 14:46:51.292388 5268 tablet_service.cc:1514] Processing DeleteTablet for tablet aabf23754c4e4d8f9055333d73e7d7d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:46:51 UTC) from {username='slave'} at 127.0.0.1:43382
I20260619 14:46:51.293098 5505 tablet_replica.cc:330] T aabf23754c4e4d8f9055333d73e7d7d1 P e7c133f1c2a64b8d99f383b68be12333: stopping tablet replica
I20260619 14:46:51.293422 5137 tablet_service.cc:1514] Processing DeleteTablet for tablet aabf23754c4e4d8f9055333d73e7d7d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:46:51 UTC) from {username='slave'} at 127.0.0.1:37064
I20260619 14:46:51.293510 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 5075
I20260619 14:46:51.293802 5505 raft_consensus.cc:2229] T aabf23754c4e4d8f9055333d73e7d7d1 P e7c133f1c2a64b8d99f383b68be12333 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:46:51.294220 5505 raft_consensus.cc:2258] T aabf23754c4e4d8f9055333d73e7d7d1 P e7c133f1c2a64b8d99f383b68be12333 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:46:51.294766 5506 tablet_replica.cc:330] T aabf23754c4e4d8f9055333d73e7d7d1 P 9565ae1d8e994fb3a74b4ecec30e6f61: stopping tablet replica
I20260619 14:46:51.295169 5506 raft_consensus.cc:2229] T aabf23754c4e4d8f9055333d73e7d7d1 P 9565ae1d8e994fb3a74b4ecec30e6f61 [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:46:51.295567 5506 raft_consensus.cc:2258] T aabf23754c4e4d8f9055333d73e7d7d1 P 9565ae1d8e994fb3a74b4ecec30e6f61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:46:51.296340 5505 ts_tablet_manager.cc:1905] T aabf23754c4e4d8f9055333d73e7d7d1 P e7c133f1c2a64b8d99f383b68be12333: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:46:51.297545 5506 ts_tablet_manager.cc:1905] T aabf23754c4e4d8f9055333d73e7d7d1 P 9565ae1d8e994fb3a74b4ecec30e6f61: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:46:51.300261 5506 ts_tablet_manager.cc:1918] T aabf23754c4e4d8f9055333d73e7d7d1 P 9565ae1d8e994fb3a74b4ecec30e6f61: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:46:51.300462 5506 log.cc:1196] T aabf23754c4e4d8f9055333d73e7d7d1 P 9565ae1d8e994fb3a74b4ecec30e6f61: Deleting WAL directory at /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880383132139-392-0/minicluster-data/ts-2/wal/wals/aabf23754c4e4d8f9055333d73e7d7d1
I20260619 14:46:51.300695 5505 ts_tablet_manager.cc:1918] T aabf23754c4e4d8f9055333d73e7d7d1 P e7c133f1c2a64b8d99f383b68be12333: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:46:51.300822 5505 log.cc:1196] T aabf23754c4e4d8f9055333d73e7d7d1 P e7c133f1c2a64b8d99f383b68be12333: Deleting WAL directory at /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781880383132139-392-0/minicluster-data/ts-1/wal/wals/aabf23754c4e4d8f9055333d73e7d7d1
I20260619 14:46:51.301043 5506 ts_tablet_manager.cc:1939] T aabf23754c4e4d8f9055333d73e7d7d1 P 9565ae1d8e994fb3a74b4ecec30e6f61: Deleting consensus metadata
I20260619 14:46:51.301468 5505 ts_tablet_manager.cc:1939] T aabf23754c4e4d8f9055333d73e7d7d1 P e7c133f1c2a64b8d99f383b68be12333: Deleting consensus metadata
I20260619 14:46:51.303705 5002 catalog_manager.cc:4841] TS e7c133f1c2a64b8d99f383b68be12333 (127.0.98.2:41237): tablet aabf23754c4e4d8f9055333d73e7d7d1 (table test-table [id=dd6732e93a1c4129b979871672b24846]) successfully deleted
I20260619 14:46:51.303879 5001 catalog_manager.cc:4841] TS 9565ae1d8e994fb3a74b4ecec30e6f61 (127.0.98.3:35187): tablet aabf23754c4e4d8f9055333d73e7d7d1 (table test-table [id=dd6732e93a1c4129b979871672b24846]) successfully deleted
W20260619 14:46:51.305298 5002 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34839 (error 108)
I20260619 14:46:51.306612 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 5206
I20260619 14:46:51.306882 5479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:51.306528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34839 (local address 127.0.0.1:37080)
0619 14:46:51.306726 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:51.306795 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34839: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
W20260619 14:46:51.307144 5002 catalog_manager.cc:4584] TS 862fede6f5f1423a9de739dc7c93c28c (127.0.98.1:34839): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet aabf23754c4e4d8f9055333d73e7d7d1: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34839: connect: Connection refused (error 111)
I20260619 14:46:51.315798 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 5337
I20260619 14:46:51.323735 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 4984
2026-06-19T14:46:51Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1115 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260619 14:46:51.346030 392 mini_oidc.cc:152] Starting JWKS server
I20260619 14:46:51.347728 392 webserver.cc:460] Webserver started at https://127.0.0.1:40833/ using document root <none> and password file <none>
I20260619 14:46:51.347882 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 14:46:51.348192 392 webserver.cc:460] Webserver started at http://127.0.0.1:43221/ using document root <none> and password file <none>
2026-06-19T14:46:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:46:51Z Disabled control of system clock
I20260619 14:46:51.358659 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781880383132139-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781880383132139-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781880383132139-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35977
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:39151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35977
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40833/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781880383132139-392-0/testchainca.cert with env {}
W20260619 14:46:51.484313 5513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:46:51.484678 5513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:46:51.484736 5513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:46:51.489095 5513 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:46:51.489183 5513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:46:51.489205 5513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:46:51.489223 5513 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:46:51.489239 5513 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 14:46:51.489256 5513 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 14:46:51.489274 5513 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40833/jwks.json
I20260619 14:46:51.495034 5513 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781880383132139-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781880383132139-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35977
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40833/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781880383132139-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5513
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781880383132139-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:46:51.496311 5513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:46:51.497383 5513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:46:51.507617 5519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:51.507741 5513 server_base.cc:1034] running on GCE node
W20260619 14:46:51.507715 5521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:51.507617 5518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:51.508987 5513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:46:51.510493 5513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:46:51.511716 5513 hybrid_clock.cc:648] HybridClock initialized: now 1781880411511675 us; error 64 us; skew 500 ppm
I20260619 14:46:51.514074 5513 webserver.cc:460] Webserver started at http://127.0.98.62:42765/ using document root <none> and password file <none>
I20260619 14:46:51.514824 5513 fs_manager.cc:359] Metadata directory not provided
I20260619 14:46:51.514896 5513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:46:51.515134 5513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:46:51.517490 5513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781880383132139-392-0/minicluster-data/master-0/data/instance:
uuid: "b28e50923b3249df8d33db0bed2c0b57"
format_stamp: "Formatted at 2026-06-19 14:46:51 on dist-test-slave-2sb1"
server_key: "c92072ab111d7126d2c7fff6778c78ca"
server_key_iv: "5e69afc06fa47c3afd1aeae47b946ea1"
server_key_version: "encryptionkey@0"
I20260619 14:46:51.518141 5513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781880383132139-392-0/minicluster-data/master-0/wal/instance:
uuid: "b28e50923b3249df8d33db0bed2c0b57"
format_stamp: "Formatted at 2026-06-19 14:46:51 on dist-test-slave-2sb1"
server_key: "c92072ab111d7126d2c7fff6778c78ca"
server_key_iv: "5e69afc06fa47c3afd1aeae47b946ea1"
server_key_version: "encryptionkey@0"
I20260619 14:46:51.523104 5513 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260619 14:46:51.526216 5527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:51.528045 5513 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 14:46:51.528200 5513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781880383132139-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781880383132139-392-0/minicluster-data/master-0/wal
uuid: "b28e50923b3249df8d33db0bed2c0b57"
format_stamp: "Formatted at 2026-06-19 14:46:51 on dist-test-slave-2sb1"
server_key: "c92072ab111d7126d2c7fff6778c78ca"
server_key_iv: "5e69afc06fa47c3afd1aeae47b946ea1"
server_key_version: "encryptionkey@0"
I20260619 14:46:51.528430 5513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781880383132139-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781880383132139-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781880383132139-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:46:51.545658 5513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:46:51.546540 5513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:46:51.546763 5513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:46:51.556100 5513 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35977
I20260619 14:46:51.556128 5579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35977 every 8 connection(s)
I20260619 14:46:51.557617 5513 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:46:51.562394 5580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:46:51.567130 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 5513
I20260619 14:46:51.567349 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781880383132139-392-0/minicluster-data/master-0/wal/instance
I20260619 14:46:51.567680 392 external_mini_cluster.cc:1428] Setting key e30a58813b375b0cf8edd5dc5da652e0
I20260619 14:46:51.569233 5580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b28e50923b3249df8d33db0bed2c0b57: Bootstrap starting.
I20260619 14:46:51.571738 5580 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b28e50923b3249df8d33db0bed2c0b57: Neither blocks nor log segments found. Creating new log.
I20260619 14:46:51.572607 5580 log.cc:824] T 00000000000000000000000000000000 P b28e50923b3249df8d33db0bed2c0b57: Log is configured to *not* fsync() on all Append() calls
I20260619 14:46:51.576045 5580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b28e50923b3249df8d33db0bed2c0b57: No bootstrap required, opened a new log
I20260619 14:46:51.578675 5583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:51.569171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37148 (local address 127.0.98.62:35977)
0619 14:46:51.570002 (+ 831us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:51.570019 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:51.570059 (+ 40us) server_negotiation.cc:407] Connection header received
0619 14:46:51.570781 (+ 722us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:51.570801 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:51.571255 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:51.571645 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:51.572149 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:51.573371 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:51.574411 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:51.574946 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:51.575087 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:51.576486 (+ 1399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:51.576493 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:51.576508 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:46:51.577081 (+ 573us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:51.577154 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:46:51.577178 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:51.577291 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:51.577514 (+ 223us) server_negotiation.cc:299] Negotiation successful
0619 14:46:51.577855 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":159,"threads_started":1}
I20260619 14:46:51.581863 5580 raft_consensus.cc:348] T 00000000000000000000000000000000 P b28e50923b3249df8d33db0bed2c0b57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b28e50923b3249df8d33db0bed2c0b57" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35977 } }
I20260619 14:46:51.582360 5580 raft_consensus.cc:374] T 00000000000000000000000000000000 P b28e50923b3249df8d33db0bed2c0b57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:46:51.582664 5580 raft_consensus.cc:729] T 00000000000000000000000000000000 P b28e50923b3249df8d33db0bed2c0b57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b28e50923b3249df8d33db0bed2c0b57, State: Initialized, Role: FOLLOWER
I20260619 14:46:51.583335 5580 consensus_queue.cc:260] T 00000000000000000000000000000000 P b28e50923b3249df8d33db0bed2c0b57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b28e50923b3249df8d33db0bed2c0b57" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35977 } }
I20260619 14:46:51.583792 5580 raft_consensus.cc:388] T 00000000000000000000000000000000 P b28e50923b3249df8d33db0bed2c0b57 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:46:51.583843 5580 raft_consensus.cc:482] T 00000000000000000000000000000000 P b28e50923b3249df8d33db0bed2c0b57 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:46:51.583952 5580 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b28e50923b3249df8d33db0bed2c0b57 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:46:51.584039 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:46:51.585359 5580 raft_consensus.cc:504] T 00000000000000000000000000000000 P b28e50923b3249df8d33db0bed2c0b57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b28e50923b3249df8d33db0bed2c0b57" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35977 } }
I20260619 14:46:51.585852 5580 leader_election.cc:302] T 00000000000000000000000000000000 P b28e50923b3249df8d33db0bed2c0b57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b28e50923b3249df8d33db0bed2c0b57; no voters:
I20260619 14:46:51.586649 5580 leader_election.cc:288] T 00000000000000000000000000000000 P b28e50923b3249df8d33db0bed2c0b57 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:46:51.586845 5585 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b28e50923b3249df8d33db0bed2c0b57 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:46:51.587126 5585 raft_consensus.cc:686] T 00000000000000000000000000000000 P b28e50923b3249df8d33db0bed2c0b57 [term 1 LEADER]: Becoming Leader. State: Replica: b28e50923b3249df8d33db0bed2c0b57, State: Running, Role: LEADER
I20260619 14:46:51.587630 5585 consensus_queue.cc:237] T 00000000000000000000000000000000 P b28e50923b3249df8d33db0bed2c0b57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b28e50923b3249df8d33db0bed2c0b57" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35977 } }
I20260619 14:46:51.588471 5580 sys_catalog.cc:564] T 00000000000000000000000000000000 P b28e50923b3249df8d33db0bed2c0b57 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:46:51.590083 5586 sys_catalog.cc:455] T 00000000000000000000000000000000 P b28e50923b3249df8d33db0bed2c0b57 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b28e50923b3249df8d33db0bed2c0b57. Latest consensus state: current_term: 1 leader_uuid: "b28e50923b3249df8d33db0bed2c0b57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b28e50923b3249df8d33db0bed2c0b57" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35977 } } }
I20260619 14:46:51.590240 5586 sys_catalog.cc:458] T 00000000000000000000000000000000 P b28e50923b3249df8d33db0bed2c0b57 [sys.catalog]: This master's current role is: LEADER
I20260619 14:46:51.590672 5587 sys_catalog.cc:455] T 00000000000000000000000000000000 P b28e50923b3249df8d33db0bed2c0b57 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b28e50923b3249df8d33db0bed2c0b57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b28e50923b3249df8d33db0bed2c0b57" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35977 } } }
I20260619 14:46:51.590794 5587 sys_catalog.cc:458] T 00000000000000000000000000000000 P b28e50923b3249df8d33db0bed2c0b57 [sys.catalog]: This master's current role is: LEADER
I20260619 14:46:51.591075 5594 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:46:51.596174 5594 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:46:51.603892 5594 catalog_manager.cc:1347] Generated new cluster ID: d263d1f6f44a4a77aa51c1baa9f8c63c
I20260619 14:46:51.603994 5594 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:46:51.628276 5594 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:46:51.630052 5594 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:46:51.641793 5594 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b28e50923b3249df8d33db0bed2c0b57: Generated new TSK 0
I20260619 14:46:51.642647 5594 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 14:46:51.684832 5610 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260619 14:46:51.685199 5610 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 14:46:51.685595 5583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:51.665314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37150 (local address 127.0.98.62:35977)
0619 14:46:51.665749 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:51.665754 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:51.665781 (+ 27us) server_negotiation.cc:407] Connection header received
0619 14:46:51.666310 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:51.666315 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:51.666523 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:51.666666 (+ 143us) server_negotiation.cc:226] Negotiated authn=JWT
0619 14:46:51.666994 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:51.667965 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:51.669087 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:51.669501 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:51.669623 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:51.670076 (+ 453us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 14:46:51.685104 (+ 15028us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40833/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 14:46:51.685430 (+ 326us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37150: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40833/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":322}
W20260619 14:46:51.685688 5583 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37150: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40833/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260619 14:46:51.686853 5609 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:46:51.665048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35977 (local address 127.0.0.1:37150)
0619 14:46:51.665703 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:51.665721 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:51.665859 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:51.666715 (+ 856us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:51.666718 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:51.666726 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0619 14:46:51.666851 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:51.666856 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:51.668117 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:51.668121 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:51.668932 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:51.668945 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:51.669656 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:51.669681 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:51.669759 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:51.669823 (+ 64us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 14:46:51.685380 (+ 15557us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40833/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0619 14:46:51.685701 (+ 321us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:35977: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40833/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":125,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:35977: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40833/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260619 14:46:51.687804 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 5513
2026-06-19T14:46:51Z chronyd exiting
I20260619 14:46:51.726930 392 test_util.cc:175] -----------------------------------------------
I20260619 14:46:51.727030 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781880383132139-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (382 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260619 14:46:51.728134 392 mini_oidc.cc:152] Starting JWKS server
I20260619 14:46:51.729126 392 webserver.cc:460] Webserver started at https://127.0.0.1:38403/ using document root <none> and password file <none>
I20260619 14:46:51.729214 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 14:46:51.729552 392 webserver.cc:460] Webserver started at http://127.0.0.1:36431/ using document root <none> and password file <none>
2026-06-19T14:46:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:46:51Z Disabled control of system clock
I20260619 14:46:51.739804 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781880383132139-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781880383132139-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781880383132139-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41497
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:40067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41497
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38403/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781880383132139-392-0/testchainca.cert with env {}
W20260619 14:46:51.863559 5616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:46:51.863905 5616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:46:51.863961 5616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:46:51.869128 5616 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:46:51.869220 5616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:46:51.869237 5616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:46:51.869256 5616 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:46:51.869274 5616 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 14:46:51.869293 5616 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 14:46:51.869308 5616 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38403/jwks.json
I20260619 14:46:51.876144 5616 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781880383132139-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781880383132139-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41497
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41497
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38403/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781880383132139-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5616
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781880383132139-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:46:51.877421 5616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:46:51.878477 5616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:46:51.889106 5624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:51.889247 5621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:51.889247 5622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:51.889515 5616 server_base.cc:1034] running on GCE node
I20260619 14:46:51.890288 5616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:46:51.891683 5616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:46:51.892963 5616 hybrid_clock.cc:648] HybridClock initialized: now 1781880411892926 us; error 60 us; skew 500 ppm
I20260619 14:46:51.895447 5616 webserver.cc:460] Webserver started at http://127.0.98.62:34859/ using document root <none> and password file <none>
I20260619 14:46:51.896140 5616 fs_manager.cc:359] Metadata directory not provided
I20260619 14:46:51.896243 5616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:46:51.896523 5616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:46:51.898947 5616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781880383132139-392-0/minicluster-data/master-0/data/instance:
uuid: "7e84d8f4e0394ed8bedb77eb0bf17061"
format_stamp: "Formatted at 2026-06-19 14:46:51 on dist-test-slave-2sb1"
server_key: "c044157924cfa07f1850e253a4ec1353"
server_key_iv: "ff2037a429c0cca6ea3b0f6895c7e33d"
server_key_version: "encryptionkey@0"
I20260619 14:46:51.899636 5616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781880383132139-392-0/minicluster-data/master-0/wal/instance:
uuid: "7e84d8f4e0394ed8bedb77eb0bf17061"
format_stamp: "Formatted at 2026-06-19 14:46:51 on dist-test-slave-2sb1"
server_key: "c044157924cfa07f1850e253a4ec1353"
server_key_iv: "ff2037a429c0cca6ea3b0f6895c7e33d"
server_key_version: "encryptionkey@0"
I20260619 14:46:51.904017 5616 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 14:46:51.907171 5630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:51.908825 5616 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:46:51.909073 5616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781880383132139-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781880383132139-392-0/minicluster-data/master-0/wal
uuid: "7e84d8f4e0394ed8bedb77eb0bf17061"
format_stamp: "Formatted at 2026-06-19 14:46:51 on dist-test-slave-2sb1"
server_key: "c044157924cfa07f1850e253a4ec1353"
server_key_iv: "ff2037a429c0cca6ea3b0f6895c7e33d"
server_key_version: "encryptionkey@0"
I20260619 14:46:51.909243 5616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781880383132139-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781880383132139-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781880383132139-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:46:51.936712 5616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:46:51.937510 5616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:46:51.937728 5616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:46:51.945382 5616 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41497
I20260619 14:46:51.945492 5682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41497 every 8 connection(s)
I20260619 14:46:51.946470 5616 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:46:51.947266 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 5616
I20260619 14:46:51.947439 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781880383132139-392-0/minicluster-data/master-0/wal/instance
I20260619 14:46:51.947817 392 external_mini_cluster.cc:1428] Setting key ea6e3f530ee58a55327ac8798ec63979
I20260619 14:46:51.951053 5683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:46:51.958969 5683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e84d8f4e0394ed8bedb77eb0bf17061: Bootstrap starting.
I20260619 14:46:51.960187 5686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:51.949389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55854 (local address 127.0.98.62:41497)
0619 14:46:51.950127 (+ 738us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:51.950149 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:51.950198 (+ 49us) server_negotiation.cc:407] Connection header received
0619 14:46:51.951172 (+ 974us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:51.951198 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:51.951779 (+ 581us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:51.952230 (+ 451us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:51.952773 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:51.954465 (+ 1692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:51.955522 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:51.956329 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:51.956521 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:51.957800 (+ 1279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:51.957808 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:51.957828 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:46:51.958409 (+ 581us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:51.958493 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:46:51.958523 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:51.958649 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:51.958842 (+ 193us) server_negotiation.cc:299] Negotiation successful
0619 14:46:51.959160 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":121,"threads_started":1}
I20260619 14:46:51.962162 5683 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e84d8f4e0394ed8bedb77eb0bf17061: Neither blocks nor log segments found. Creating new log.
I20260619 14:46:51.963323 5683 log.cc:824] T 00000000000000000000000000000000 P 7e84d8f4e0394ed8bedb77eb0bf17061: Log is configured to *not* fsync() on all Append() calls
I20260619 14:46:51.964581 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:46:51.966430 5683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e84d8f4e0394ed8bedb77eb0bf17061: No bootstrap required, opened a new log
I20260619 14:46:51.970430 5683 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e84d8f4e0394ed8bedb77eb0bf17061 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e84d8f4e0394ed8bedb77eb0bf17061" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41497 } }
I20260619 14:46:51.970709 5683 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7e84d8f4e0394ed8bedb77eb0bf17061 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:46:51.970746 5683 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e84d8f4e0394ed8bedb77eb0bf17061 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e84d8f4e0394ed8bedb77eb0bf17061, State: Initialized, Role: FOLLOWER
I20260619 14:46:51.971355 5683 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e84d8f4e0394ed8bedb77eb0bf17061 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e84d8f4e0394ed8bedb77eb0bf17061" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41497 } }
I20260619 14:46:51.971536 5683 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7e84d8f4e0394ed8bedb77eb0bf17061 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:46:51.971625 5683 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e84d8f4e0394ed8bedb77eb0bf17061 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:46:51.971755 5683 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e84d8f4e0394ed8bedb77eb0bf17061 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:46:51.972945 5683 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e84d8f4e0394ed8bedb77eb0bf17061 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e84d8f4e0394ed8bedb77eb0bf17061" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41497 } }
I20260619 14:46:51.973333 5683 leader_election.cc:302] T 00000000000000000000000000000000 P 7e84d8f4e0394ed8bedb77eb0bf17061 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e84d8f4e0394ed8bedb77eb0bf17061; no voters:
I20260619 14:46:51.973867 5688 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e84d8f4e0394ed8bedb77eb0bf17061 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:46:51.973865 5683 leader_election.cc:288] T 00000000000000000000000000000000 P 7e84d8f4e0394ed8bedb77eb0bf17061 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:46:51.974396 5688 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7e84d8f4e0394ed8bedb77eb0bf17061 [term 1 LEADER]: Becoming Leader. State: Replica: 7e84d8f4e0394ed8bedb77eb0bf17061, State: Running, Role: LEADER
I20260619 14:46:51.974750 5688 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e84d8f4e0394ed8bedb77eb0bf17061 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e84d8f4e0394ed8bedb77eb0bf17061" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41497 } }
I20260619 14:46:51.976167 5683 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e84d8f4e0394ed8bedb77eb0bf17061 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:46:51.976605 5690 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e84d8f4e0394ed8bedb77eb0bf17061 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7e84d8f4e0394ed8bedb77eb0bf17061" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e84d8f4e0394ed8bedb77eb0bf17061" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41497 } } }
I20260619 14:46:51.976795 5690 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e84d8f4e0394ed8bedb77eb0bf17061 [sys.catalog]: This master's current role is: LEADER
I20260619 14:46:51.976893 5689 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e84d8f4e0394ed8bedb77eb0bf17061 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e84d8f4e0394ed8bedb77eb0bf17061. Latest consensus state: current_term: 1 leader_uuid: "7e84d8f4e0394ed8bedb77eb0bf17061" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e84d8f4e0394ed8bedb77eb0bf17061" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41497 } } }
I20260619 14:46:51.977104 5689 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e84d8f4e0394ed8bedb77eb0bf17061 [sys.catalog]: This master's current role is: LEADER
I20260619 14:46:51.977229 5692 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:46:51.981024 5692 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:46:51.987869 5692 catalog_manager.cc:1347] Generated new cluster ID: 0195a271625c4abfb187367333cbc581
I20260619 14:46:51.988070 5692 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:46:52.006314 5692 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:46:52.007936 5692 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:46:52.014933 5692 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7e84d8f4e0394ed8bedb77eb0bf17061: Generated new TSK 0
I20260619 14:46:52.015960 5692 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260619 14:46:52.047116 5713 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260619 14:46:52.047243 5713 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260619 14:46:52.048028 5686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:52.028289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55868 (local address 127.0.98.62:41497)
0619 14:46:52.028452 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:52.028457 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:52.028666 (+ 209us) server_negotiation.cc:407] Connection header received
0619 14:46:52.028882 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:52.028886 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:52.028949 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:52.029096 (+ 147us) server_negotiation.cc:226] Negotiated authn=JWT
0619 14:46:52.029399 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:52.030251 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:52.031278 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:52.031622 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:52.031732 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:52.032122 (+ 390us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 14:46:52.047538 (+ 15416us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38403/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 14:46:52.047866 (+ 328us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55868: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38403/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":58}
W20260619 14:46:52.048110 5686 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55868: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38403/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260619 14:46:52.048399 5712 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:46:52.028109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41497 (local address 127.0.0.1:55868)
0619 14:46:52.028539 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:52.028553 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:52.028724 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:52.029142 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:52.029145 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:52.029153 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0619 14:46:52.029272 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:52.029278 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:52.030401 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:52.030405 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:52.031151 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:52.031160 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:52.031753 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:52.031775 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:52.031860 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:52.031889 (+ 29us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 14:46:52.047815 (+ 15926us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38403/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0619 14:46:52.048197 (+ 382us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:41497: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38403/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":144,"threads_started":1}
I20260619 14:46:52.049428 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 5616
2026-06-19T14:46:52Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (362 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260619 14:46:52.091013 392 mini_oidc.cc:152] Starting JWKS server
I20260619 14:46:52.091961 392 webserver.cc:460] Webserver started at https://127.0.0.1:45097/ using document root <none> and password file <none>
I20260619 14:46:52.092108 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260619 14:46:52.092386 392 webserver.cc:460] Webserver started at http://127.0.0.1:39339/ using document root <none> and password file <none>
2026-06-19T14:46:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:46:52Z Disabled control of system clock
I20260619 14:46:52.102787 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781880383132139-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781880383132139-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781880383132139-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40989
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:38247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40989
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45097/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 14:46:52.232730 5719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:46:52.233234 5719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:46:52.233353 5719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:46:52.238246 5719 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:46:52.238354 5719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:46:52.238371 5719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:46:52.238390 5719 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:46:52.238406 5719 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 14:46:52.238605 5719 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 14:46:52.238651 5719 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45097/jwks.json
I20260619 14:46:52.246766 5719 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781880383132139-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781880383132139-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40989
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45097/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781880383132139-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:46:52.249861 5719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:46:52.253513 5719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:46:52.265532 5727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:52.265442 5724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:52.267001 5719 server_base.cc:1034] running on GCE node
W20260619 14:46:52.267031 5725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:52.267824 5719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:46:52.269707 5719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:46:52.271001 5719 hybrid_clock.cc:648] HybridClock initialized: now 1781880412270972 us; error 54 us; skew 500 ppm
I20260619 14:46:52.273590 5719 webserver.cc:460] Webserver started at http://127.0.98.62:38933/ using document root <none> and password file <none>
I20260619 14:46:52.274277 5719 fs_manager.cc:359] Metadata directory not provided
I20260619 14:46:52.274365 5719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:46:52.274616 5719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:46:52.277176 5719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781880383132139-392-0/minicluster-data/master-0/data/instance:
uuid: "d4f38db5f32d467f86c0a5079c453457"
format_stamp: "Formatted at 2026-06-19 14:46:52 on dist-test-slave-2sb1"
server_key: "fdb523aa8634cfa9e2088a99b065b7cb"
server_key_iv: "98f719da954811ebc10b0bf318419bbb"
server_key_version: "encryptionkey@0"
I20260619 14:46:52.277858 5719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781880383132139-392-0/minicluster-data/master-0/wal/instance:
uuid: "d4f38db5f32d467f86c0a5079c453457"
format_stamp: "Formatted at 2026-06-19 14:46:52 on dist-test-slave-2sb1"
server_key: "fdb523aa8634cfa9e2088a99b065b7cb"
server_key_iv: "98f719da954811ebc10b0bf318419bbb"
server_key_version: "encryptionkey@0"
I20260619 14:46:52.282397 5719 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 14:46:52.285647 5733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:52.287472 5719 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:46:52.287619 5719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781880383132139-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781880383132139-392-0/minicluster-data/master-0/wal
uuid: "d4f38db5f32d467f86c0a5079c453457"
format_stamp: "Formatted at 2026-06-19 14:46:52 on dist-test-slave-2sb1"
server_key: "fdb523aa8634cfa9e2088a99b065b7cb"
server_key_iv: "98f719da954811ebc10b0bf318419bbb"
server_key_version: "encryptionkey@0"
I20260619 14:46:52.287751 5719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781880383132139-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781880383132139-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781880383132139-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:46:52.308389 5719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:46:52.309149 5719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:46:52.309329 5719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:46:52.317523 5719 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40989
I20260619 14:46:52.317523 5785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40989 every 8 connection(s)
I20260619 14:46:52.318706 5719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:46:52.322472 5786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:46:52.327318 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 5719
I20260619 14:46:52.327606 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781880383132139-392-0/minicluster-data/master-0/wal/instance
I20260619 14:46:52.327884 392 external_mini_cluster.cc:1428] Setting key d79f0980ac1ee583c822a0b39a4f9de1
I20260619 14:46:52.329598 5786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4f38db5f32d467f86c0a5079c453457: Bootstrap starting.
I20260619 14:46:52.332044 5786 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d4f38db5f32d467f86c0a5079c453457: Neither blocks nor log segments found. Creating new log.
I20260619 14:46:52.333037 5786 log.cc:824] T 00000000000000000000000000000000 P d4f38db5f32d467f86c0a5079c453457: Log is configured to *not* fsync() on all Append() calls
I20260619 14:46:52.336002 5786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4f38db5f32d467f86c0a5079c453457: No bootstrap required, opened a new log
I20260619 14:46:52.339504 5789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:52.329532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33108 (local address 127.0.98.62:40989)
0619 14:46:52.330342 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:52.330361 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:52.330402 (+ 41us) server_negotiation.cc:407] Connection header received
0619 14:46:52.331159 (+ 757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:52.331179 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:52.331633 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:52.332075 (+ 442us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:52.332813 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:52.333876 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:52.335018 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:52.335708 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:52.335880 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:52.337092 (+ 1212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:52.337099 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:52.337117 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:46:52.337675 (+ 558us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:52.337756 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:46:52.337787 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:52.337904 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:52.338110 (+ 206us) server_negotiation.cc:299] Negotiation successful
0619 14:46:52.338426 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":231,"threads_started":1}
I20260619 14:46:52.341028 5786 raft_consensus.cc:348] T 00000000000000000000000000000000 P d4f38db5f32d467f86c0a5079c453457 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4f38db5f32d467f86c0a5079c453457" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40989 } }
I20260619 14:46:52.341288 5786 raft_consensus.cc:374] T 00000000000000000000000000000000 P d4f38db5f32d467f86c0a5079c453457 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:46:52.341323 5786 raft_consensus.cc:729] T 00000000000000000000000000000000 P d4f38db5f32d467f86c0a5079c453457 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4f38db5f32d467f86c0a5079c453457, State: Initialized, Role: FOLLOWER
I20260619 14:46:52.341886 5786 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4f38db5f32d467f86c0a5079c453457 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4f38db5f32d467f86c0a5079c453457" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40989 } }
I20260619 14:46:52.342095 5786 raft_consensus.cc:388] T 00000000000000000000000000000000 P d4f38db5f32d467f86c0a5079c453457 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:46:52.342155 5786 raft_consensus.cc:482] T 00000000000000000000000000000000 P d4f38db5f32d467f86c0a5079c453457 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:46:52.342242 5786 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d4f38db5f32d467f86c0a5079c453457 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:46:52.343469 5786 raft_consensus.cc:504] T 00000000000000000000000000000000 P d4f38db5f32d467f86c0a5079c453457 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4f38db5f32d467f86c0a5079c453457" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40989 } }
I20260619 14:46:52.343636 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:46:52.343883 5786 leader_election.cc:302] T 00000000000000000000000000000000 P d4f38db5f32d467f86c0a5079c453457 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d4f38db5f32d467f86c0a5079c453457; no voters:
I20260619 14:46:52.344399 5786 leader_election.cc:288] T 00000000000000000000000000000000 P d4f38db5f32d467f86c0a5079c453457 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:46:52.344542 5791 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d4f38db5f32d467f86c0a5079c453457 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:46:52.344826 5791 raft_consensus.cc:686] T 00000000000000000000000000000000 P d4f38db5f32d467f86c0a5079c453457 [term 1 LEADER]: Becoming Leader. State: Replica: d4f38db5f32d467f86c0a5079c453457, State: Running, Role: LEADER
I20260619 14:46:52.345211 5791 consensus_queue.cc:237] T 00000000000000000000000000000000 P d4f38db5f32d467f86c0a5079c453457 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4f38db5f32d467f86c0a5079c453457" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40989 } }
I20260619 14:46:52.345862 5786 sys_catalog.cc:564] T 00000000000000000000000000000000 P d4f38db5f32d467f86c0a5079c453457 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:46:52.347286 5792 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4f38db5f32d467f86c0a5079c453457 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d4f38db5f32d467f86c0a5079c453457" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4f38db5f32d467f86c0a5079c453457" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40989 } } }
I20260619 14:46:52.347608 5792 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4f38db5f32d467f86c0a5079c453457 [sys.catalog]: This master's current role is: LEADER
I20260619 14:46:52.348055 5800 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:46:52.348055 5793 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4f38db5f32d467f86c0a5079c453457 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4f38db5f32d467f86c0a5079c453457. Latest consensus state: current_term: 1 leader_uuid: "d4f38db5f32d467f86c0a5079c453457" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4f38db5f32d467f86c0a5079c453457" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40989 } } }
I20260619 14:46:52.348214 5793 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4f38db5f32d467f86c0a5079c453457 [sys.catalog]: This master's current role is: LEADER
I20260619 14:46:52.351373 5800 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:46:52.358960 5800 catalog_manager.cc:1347] Generated new cluster ID: 30e0e46fb4ca4264baedb315ee80fc59
I20260619 14:46:52.359074 5800 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:46:52.390699 5800 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:46:52.392763 5800 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:46:52.403978 5800 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d4f38db5f32d467f86c0a5079c453457: Generated new TSK 0
I20260619 14:46:52.405036 5800 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:46:52.428709 5789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:52.422243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33124 (local address 127.0.98.62:40989)
0619 14:46:52.422530 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:52.422534 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:52.422675 (+ 141us) server_negotiation.cc:407] Connection header received
0619 14:46:52.422939 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:52.422942 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:52.422999 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:52.423106 (+ 107us) server_negotiation.cc:226] Negotiated authn=JWT
0619 14:46:52.423457 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:52.424124 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:52.425695 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:52.425991 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:52.426138 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:52.426665 (+ 527us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 14:46:52.428229 (+ 1564us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45097/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 14:46:52.428525 (+ 296us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33124: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45097/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":202}
W20260619 14:46:52.428792 5789 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33124: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45097/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260619 14:46:52.428860 5815 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:46:52.422136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40989 (local address 127.0.0.1:33124)
0619 14:46:52.422550 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:52.422567 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:52.422694 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:52.423145 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:52.423148 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:52.423156 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0619 14:46:52.423278 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:52.423285 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:52.424292 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:52.424296 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:52.425497 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:52.425512 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:52.426244 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:52.426279 (+ 35us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:52.426394 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:52.426432 (+ 38us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 14:46:52.428444 (+ 2012us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45097/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0619 14:46:52.428620 (+ 176us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:40989: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45097/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":123,"threads_started":1}
I20260619 14:46:52.431875 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 5719
E20260619 14:46:52.441905 5816 webserver.cc:533] Webserver: sslize failed:
E20260619 14:46:52.442009 5816 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-06-19T14:46:52Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (381 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-19T14:46:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:46:52Z Disabled control of system clock
I20260619 14:46:52.482753 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781880383132139-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781880383132139-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781880383132139-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38393
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:39943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38393
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781880383132139-392-0/testchainca.cert with env {}
W20260619 14:46:52.619174 5820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:46:52.619886 5820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:46:52.619963 5820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:46:52.625335 5820 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:46:52.625428 5820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:46:52.625445 5820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:46:52.625463 5820 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:46:52.625479 5820 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260619 14:46:52.625497 5820 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260619 14:46:52.625511 5820 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260619 14:46:52.631539 5820 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781880383132139-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781880383132139-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38393
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781880383132139-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5820
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781880383132139-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:46:52.632707 5820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:46:52.633728 5820 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:46:52.643486 5826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:52.643582 5825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:52.643577 5820 server_base.cc:1034] running on GCE node
W20260619 14:46:52.643486 5828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:52.644469 5820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:46:52.645730 5820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:46:52.647002 5820 hybrid_clock.cc:648] HybridClock initialized: now 1781880412646969 us; error 47 us; skew 500 ppm
I20260619 14:46:52.649415 5820 webserver.cc:460] Webserver started at http://127.0.98.62:42631/ using document root <none> and password file <none>
I20260619 14:46:52.650051 5820 fs_manager.cc:359] Metadata directory not provided
I20260619 14:46:52.650122 5820 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:46:52.650352 5820 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:46:52.652631 5820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781880383132139-392-0/minicluster-data/master-0/data/instance:
uuid: "2e2638b2573148dfbb0c63a617a0cf42"
format_stamp: "Formatted at 2026-06-19 14:46:52 on dist-test-slave-2sb1"
server_key: "8d627eeef9dd8128c04bc501a5275e2a"
server_key_iv: "bd1c3ba2796d6fdb783eca994e4d97d7"
server_key_version: "encryptionkey@0"
I20260619 14:46:52.653368 5820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781880383132139-392-0/minicluster-data/master-0/wal/instance:
uuid: "2e2638b2573148dfbb0c63a617a0cf42"
format_stamp: "Formatted at 2026-06-19 14:46:52 on dist-test-slave-2sb1"
server_key: "8d627eeef9dd8128c04bc501a5275e2a"
server_key_iv: "bd1c3ba2796d6fdb783eca994e4d97d7"
server_key_version: "encryptionkey@0"
I20260619 14:46:52.657805 5820 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 14:46:52.660877 5834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:52.662642 5820 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 14:46:52.662829 5820 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781880383132139-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781880383132139-392-0/minicluster-data/master-0/wal
uuid: "2e2638b2573148dfbb0c63a617a0cf42"
format_stamp: "Formatted at 2026-06-19 14:46:52 on dist-test-slave-2sb1"
server_key: "8d627eeef9dd8128c04bc501a5275e2a"
server_key_iv: "bd1c3ba2796d6fdb783eca994e4d97d7"
server_key_version: "encryptionkey@0"
I20260619 14:46:52.662978 5820 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781880383132139-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781880383132139-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781880383132139-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:46:52.684482 5820 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:46:52.685316 5820 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:46:52.685541 5820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:46:52.693431 5820 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38393
I20260619 14:46:52.693482 5886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38393 every 8 connection(s)
I20260619 14:46:52.694510 5820 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:46:52.698447 5887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:46:52.701908 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 5820
I20260619 14:46:52.702019 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781880383132139-392-0/minicluster-data/master-0/wal/instance
I20260619 14:46:52.702425 392 external_mini_cluster.cc:1428] Setting key a74854c4d3f7ab02ea61ef2b8f0d7400
I20260619 14:46:52.705741 5887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e2638b2573148dfbb0c63a617a0cf42: Bootstrap starting.
I20260619 14:46:52.708456 5887 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e2638b2573148dfbb0c63a617a0cf42: Neither blocks nor log segments found. Creating new log.
I20260619 14:46:52.709638 5887 log.cc:824] T 00000000000000000000000000000000 P 2e2638b2573148dfbb0c63a617a0cf42: Log is configured to *not* fsync() on all Append() calls
I20260619 14:46:52.712507 5887 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e2638b2573148dfbb0c63a617a0cf42: No bootstrap required, opened a new log
I20260619 14:46:52.714464 5890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:52.704301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39534 (local address 127.0.98.62:38393)
0619 14:46:52.705294 (+ 993us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:52.705316 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:52.705361 (+ 45us) server_negotiation.cc:407] Connection header received
0619 14:46:52.706179 (+ 818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:52.706200 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:52.706715 (+ 515us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:52.707067 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:52.707466 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:52.708556 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:52.710059 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:52.711075 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:52.711258 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:52.712201 (+ 943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:52.712208 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:52.712231 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:46:52.712786 (+ 555us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:52.712865 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:46:52.712896 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:52.713198 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:52.713327 (+ 129us) server_negotiation.cc:299] Negotiation successful
0619 14:46:52.713650 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":178,"threads_started":1}
I20260619 14:46:52.717873 5887 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e2638b2573148dfbb0c63a617a0cf42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e2638b2573148dfbb0c63a617a0cf42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38393 } }
I20260619 14:46:52.718904 5887 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2e2638b2573148dfbb0c63a617a0cf42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:46:52.719056 5887 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e2638b2573148dfbb0c63a617a0cf42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e2638b2573148dfbb0c63a617a0cf42, State: Initialized, Role: FOLLOWER
I20260619 14:46:52.719558 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:46:52.719965 5887 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e2638b2573148dfbb0c63a617a0cf42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e2638b2573148dfbb0c63a617a0cf42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38393 } }
I20260619 14:46:52.720394 5887 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2e2638b2573148dfbb0c63a617a0cf42 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:46:52.720510 5887 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e2638b2573148dfbb0c63a617a0cf42 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:46:52.720647 5887 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e2638b2573148dfbb0c63a617a0cf42 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:46:52.722009 5887 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e2638b2573148dfbb0c63a617a0cf42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e2638b2573148dfbb0c63a617a0cf42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38393 } }
I20260619 14:46:52.722465 5887 leader_election.cc:302] T 00000000000000000000000000000000 P 2e2638b2573148dfbb0c63a617a0cf42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e2638b2573148dfbb0c63a617a0cf42; no voters:
I20260619 14:46:52.722863 5887 leader_election.cc:288] T 00000000000000000000000000000000 P 2e2638b2573148dfbb0c63a617a0cf42 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:46:52.723071 5892 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e2638b2573148dfbb0c63a617a0cf42 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:46:52.723495 5892 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2e2638b2573148dfbb0c63a617a0cf42 [term 1 LEADER]: Becoming Leader. State: Replica: 2e2638b2573148dfbb0c63a617a0cf42, State: Running, Role: LEADER
I20260619 14:46:52.723907 5892 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e2638b2573148dfbb0c63a617a0cf42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e2638b2573148dfbb0c63a617a0cf42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38393 } }
I20260619 14:46:52.724359 5887 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e2638b2573148dfbb0c63a617a0cf42 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:46:52.725910 5894 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e2638b2573148dfbb0c63a617a0cf42 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e2638b2573148dfbb0c63a617a0cf42. Latest consensus state: current_term: 1 leader_uuid: "2e2638b2573148dfbb0c63a617a0cf42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e2638b2573148dfbb0c63a617a0cf42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38393 } } }
I20260619 14:46:52.725909 5893 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e2638b2573148dfbb0c63a617a0cf42 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2e2638b2573148dfbb0c63a617a0cf42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e2638b2573148dfbb0c63a617a0cf42" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38393 } } }
I20260619 14:46:52.726085 5894 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e2638b2573148dfbb0c63a617a0cf42 [sys.catalog]: This master's current role is: LEADER
I20260619 14:46:52.726085 5893 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e2638b2573148dfbb0c63a617a0cf42 [sys.catalog]: This master's current role is: LEADER
W20260619 14:46:52.729301 5907 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2e2638b2573148dfbb0c63a617a0cf42: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:46:52.729420 5907 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:46:52.729492 5906 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:46:52.730271 5906 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:46:52.737627 5906 catalog_manager.cc:1347] Generated new cluster ID: d7c8229cbc6140548217bf3b229d097c
I20260619 14:46:52.737735 5906 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:46:52.748248 5906 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:46:52.749351 5906 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:46:52.762750 5906 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2e2638b2573148dfbb0c63a617a0cf42: Generated new TSK 0
I20260619 14:46:52.763742 5906 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:46:53.215772 5890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:52.782600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39536 (local address 127.0.98.62:38393)
0619 14:46:52.782749 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:52.782755 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:52.782953 (+ 198us) server_negotiation.cc:407] Connection header received
0619 14:46:52.783354 (+ 401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:52.783358 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:52.783426 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:52.783575 (+ 149us) server_negotiation.cc:226] Negotiated authn=JWT
0619 14:46:52.783833 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:52.784413 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:52.785379 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:52.785621 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:52.785736 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:52.786070 (+ 334us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0619 14:46:53.214816 (+428746us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 14:46:53.215652 (+ 836us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39536: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":60}
W20260619 14:46:53.215750 5916 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:46:52.782435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38393 (local address 127.0.0.1:39536)
0619 14:46:52.782847 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:52.782860 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:52.783150 (+ 290us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:52.783572 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:52.783575 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:52.783591 (+ 16us) client_negotiation.cc:189] Negotiated authn=JWT
0619 14:46:52.783713 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:52.783719 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:52.784533 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:52.784538 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:52.785237 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:52.785245 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:52.785726 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:52.785755 (+ 29us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:52.785854 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:52.785879 (+ 25us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0619 14:46:53.215329 (+429450us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0619 14:46:53.215536 (+ 207us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:38393: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":377,"threads_started":1}
W20260619 14:46:53.215852 5890 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39536: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
I20260619 14:46:53.216797 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 5820
2026-06-19T14:46:53Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (763 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (142 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (139 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880383132139-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:46:53 dist-test-slave-2sb1 krb5kdc[5928](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:46:53 dist-test-slave-2sb1 krb5kdc[5928](info): set up 2 sockets
Jun 19 14:46:53 dist-test-slave-2sb1 krb5kdc[5928](info): commencing operation
krb5kdc: starting...
W20260619 14:46:54.613957 392 mini_kdc.cc:121] Time spent starting KDC: real 1.096s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880383132139-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880383132139-392-0/krb5kdc/test-user.keytab.
Jun 19 14:46:54 dist-test-slave-2sb1 krb5kdc[5928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880414, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:46:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:46:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 14:46:54.757112 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880383132139-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880383132139-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880383132139-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43381
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43381
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:46:54.875694 5941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:46:54.876041 5941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:46:54.876124 5941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:46:54.880486 5941 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:46:54.880628 5941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:46:54.880658 5941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:46:54.880708 5941 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:46:54.880749 5941 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:46:54.886700 5941 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880383132139-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880383132139-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43381
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43381
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5941
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880383132139-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:46:54.887919 5941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:46:54.889206 5941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:46:54.898296 5947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:54.898296 5946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:54.898386 5949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:54.898456 5941 server_base.cc:1034] running on GCE node
I20260619 14:46:54.899314 5941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:46:54.900600 5941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:46:54.901865 5941 hybrid_clock.cc:648] HybridClock initialized: now 1781880414901840 us; error 50 us; skew 500 ppm
Jun 19 14:46:54 dist-test-slave-2sb1 krb5kdc[5928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880414, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:46:54.909914 5941 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 14:46:54.911033 5941 webserver.cc:460] Webserver started at http://127.0.98.62:40915/ using document root <none> and password file <none>
I20260619 14:46:54.911680 5941 fs_manager.cc:359] Metadata directory not provided
I20260619 14:46:54.911788 5941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:46:54.912035 5941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:46:54.914732 5941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880383132139-392-0/minicluster-data/master-0/data/instance:
uuid: "b4617bad716541428ea30fe99cdda1dc"
format_stamp: "Formatted at 2026-06-19 14:46:54 on dist-test-slave-2sb1"
server_key: "56c00ff61cb7ea920ccd55914ac993fe"
server_key_iv: "9bc639dfb911c28f417faf24ab557156"
server_key_version: "encryptionkey@0"
I20260619 14:46:54.915486 5941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880383132139-392-0/minicluster-data/master-0/wal/instance:
uuid: "b4617bad716541428ea30fe99cdda1dc"
format_stamp: "Formatted at 2026-06-19 14:46:54 on dist-test-slave-2sb1"
server_key: "56c00ff61cb7ea920ccd55914ac993fe"
server_key_iv: "9bc639dfb911c28f417faf24ab557156"
server_key_version: "encryptionkey@0"
I20260619 14:46:54.920125 5941 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260619 14:46:54.923507 5956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:54.925361 5941 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:46:54.925570 5941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880383132139-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880383132139-392-0/minicluster-data/master-0/wal
uuid: "b4617bad716541428ea30fe99cdda1dc"
format_stamp: "Formatted at 2026-06-19 14:46:54 on dist-test-slave-2sb1"
server_key: "56c00ff61cb7ea920ccd55914ac993fe"
server_key_iv: "9bc639dfb911c28f417faf24ab557156"
server_key_version: "encryptionkey@0"
I20260619 14:46:54.925710 5941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880383132139-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880383132139-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880383132139-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:46:54.953926 5941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:46:54.957587 5941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:46:54.957818 5941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:46:54.965515 5941 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43381
I20260619 14:46:54.965794 6008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43381 every 8 connection(s)
I20260619 14:46:54.966657 5941 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:46:54.969679 6009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:46:54.975023 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 5941
I20260619 14:46:54.975174 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880383132139-392-0/minicluster-data/master-0/wal/instance
I20260619 14:46:54.975457 392 external_mini_cluster.cc:1428] Setting key 7cea25dc369dc0b826e77fbb60e3b9d4
I20260619 14:46:54.975837 6009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4617bad716541428ea30fe99cdda1dc: Bootstrap starting.
I20260619 14:46:54.978521 6009 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b4617bad716541428ea30fe99cdda1dc: Neither blocks nor log segments found. Creating new log.
I20260619 14:46:54.979308 6009 log.cc:824] T 00000000000000000000000000000000 P b4617bad716541428ea30fe99cdda1dc: Log is configured to *not* fsync() on all Append() calls
I20260619 14:46:54.981369 6009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4617bad716541428ea30fe99cdda1dc: No bootstrap required, opened a new log
Jun 19 14:46:54 dist-test-slave-2sb1 krb5kdc[5928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880414, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:46:54.985487 6009 raft_consensus.cc:348] T 00000000000000000000000000000000 P b4617bad716541428ea30fe99cdda1dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4617bad716541428ea30fe99cdda1dc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43381 } }
I20260619 14:46:54.985786 6009 raft_consensus.cc:374] T 00000000000000000000000000000000 P b4617bad716541428ea30fe99cdda1dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:46:54.985857 6009 raft_consensus.cc:729] T 00000000000000000000000000000000 P b4617bad716541428ea30fe99cdda1dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4617bad716541428ea30fe99cdda1dc, State: Initialized, Role: FOLLOWER
I20260619 14:46:54.986402 6009 consensus_queue.cc:260] T 00000000000000000000000000000000 P b4617bad716541428ea30fe99cdda1dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4617bad716541428ea30fe99cdda1dc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43381 } }
I20260619 14:46:54.986557 6009 raft_consensus.cc:388] T 00000000000000000000000000000000 P b4617bad716541428ea30fe99cdda1dc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:46:54.986635 6009 raft_consensus.cc:482] T 00000000000000000000000000000000 P b4617bad716541428ea30fe99cdda1dc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:46:54.986750 6009 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b4617bad716541428ea30fe99cdda1dc [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:46:54.987726 6009 raft_consensus.cc:504] T 00000000000000000000000000000000 P b4617bad716541428ea30fe99cdda1dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4617bad716541428ea30fe99cdda1dc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43381 } }
I20260619 14:46:54.988107 6009 leader_election.cc:302] T 00000000000000000000000000000000 P b4617bad716541428ea30fe99cdda1dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b4617bad716541428ea30fe99cdda1dc; no voters:
I20260619 14:46:54.988510 6009 leader_election.cc:288] T 00000000000000000000000000000000 P b4617bad716541428ea30fe99cdda1dc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:46:54.988627 6014 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b4617bad716541428ea30fe99cdda1dc [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:46:54.988904 6014 raft_consensus.cc:686] T 00000000000000000000000000000000 P b4617bad716541428ea30fe99cdda1dc [term 1 LEADER]: Becoming Leader. State: Replica: b4617bad716541428ea30fe99cdda1dc, State: Running, Role: LEADER
I20260619 14:46:54.989271 6014 consensus_queue.cc:237] T 00000000000000000000000000000000 P b4617bad716541428ea30fe99cdda1dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4617bad716541428ea30fe99cdda1dc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43381 } }
I20260619 14:46:54.990010 6009 sys_catalog.cc:564] T 00000000000000000000000000000000 P b4617bad716541428ea30fe99cdda1dc [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:46:54.992074 6012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:54.976990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47548 (local address 127.0.98.62:43381)
0619 14:46:54.977507 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:54.977529 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:54.977577 (+ 48us) server_negotiation.cc:407] Connection header received
0619 14:46:54.978354 (+ 777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:54.978373 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:54.978800 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:54.979181 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:54.980062 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:54.981115 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:54.982215 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:54.982775 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:54.982993 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:54.986050 (+ 3057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:54.986060 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:54.986082 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:54.986116 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:54.988180 (+ 2064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:54.988838 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:54.988844 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:54.988851 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:54.989001 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:54.989326 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:54.989329 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:54.989331 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:54.989989 (+ 658us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:54.990180 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:54.990682 (+ 502us) server_negotiation.cc:299] Negotiation successful
0619 14:46:54.991035 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":163,"threads_started":1}
I20260619 14:46:54.993275 6016 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4617bad716541428ea30fe99cdda1dc [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4617bad716541428ea30fe99cdda1dc. Latest consensus state: current_term: 1 leader_uuid: "b4617bad716541428ea30fe99cdda1dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4617bad716541428ea30fe99cdda1dc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43381 } } }
I20260619 14:46:54.993528 6016 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4617bad716541428ea30fe99cdda1dc [sys.catalog]: This master's current role is: LEADER
I20260619 14:46:54.993574 6015 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4617bad716541428ea30fe99cdda1dc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b4617bad716541428ea30fe99cdda1dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4617bad716541428ea30fe99cdda1dc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43381 } } }
I20260619 14:46:54.993676 6015 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4617bad716541428ea30fe99cdda1dc [sys.catalog]: This master's current role is: LEADER
W20260619 14:46:54.995584 6028 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b4617bad716541428ea30fe99cdda1dc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:46:54.995664 6028 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:46:54.995749 6030 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:46:54.996469 6030 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:46:55.002796 6030 catalog_manager.cc:1347] Generated new cluster ID: b20fa04de2b3440f8c0d2535cf26a94a
I20260619 14:46:55.003086 6030 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:46:55.013546 6030 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:46:55.014693 6030 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:46:55.022125 6030 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b4617bad716541428ea30fe99cdda1dc: Generated new TSK 0
I20260619 14:46:55.022975 6030 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880383132139-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880383132139-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 14:46:55.061391 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880383132139-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880383132139-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880383132139-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880383132139-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43381
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880383132139-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:46:55.208864 6035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:46:55.209237 6035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:46:55.209306 6035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:46:55.213861 6035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:46:55.213948 6035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:46:55.214016 6035 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 14:46:55.220359 6035 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880383132139-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880383132139-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880383132139-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880383132139-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43381
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6035
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880383132139-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:46:55.222240 6035 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:46:55.223517 6035 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:46:55.235035 6040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:55.235208 6041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:55.235328 6035 server_base.cc:1034] running on GCE node
W20260619 14:46:55.235213 6043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:55.236094 6035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:46:55.236817 6035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:46:55.238003 6035 hybrid_clock.cc:648] HybridClock initialized: now 1781880415237981 us; error 34 us; skew 500 ppm
Jun 19 14:46:55 dist-test-slave-2sb1 krb5kdc[5928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880415, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:46:55.241927 6035 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 14:46:55.242960 6035 webserver.cc:460] Webserver started at http://127.0.98.1:40677/ using document root <none> and password file <none>
I20260619 14:46:55.243685 6035 fs_manager.cc:359] Metadata directory not provided
I20260619 14:46:55.243794 6035 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:46:55.244015 6035 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:46:55.246286 6035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880383132139-392-0/minicluster-data/ts-0/data/instance:
uuid: "5c29e6d0c91e4a27abf53fe36ad4ba28"
format_stamp: "Formatted at 2026-06-19 14:46:55 on dist-test-slave-2sb1"
server_key: "f3d6e57d17f2dbbb8021ab6193f43d76"
server_key_iv: "af3c2fab706c0f0be52eeed9fafcffd0"
server_key_version: "encryptionkey@0"
I20260619 14:46:55.246901 6035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880383132139-392-0/minicluster-data/ts-0/wal/instance:
uuid: "5c29e6d0c91e4a27abf53fe36ad4ba28"
format_stamp: "Formatted at 2026-06-19 14:46:55 on dist-test-slave-2sb1"
server_key: "f3d6e57d17f2dbbb8021ab6193f43d76"
server_key_iv: "af3c2fab706c0f0be52eeed9fafcffd0"
server_key_version: "encryptionkey@0"
I20260619 14:46:55.251081 6035 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 14:46:55.254588 6050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:55.256124 6035 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 14:46:55.256304 6035 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880383132139-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880383132139-392-0/minicluster-data/ts-0/wal
uuid: "5c29e6d0c91e4a27abf53fe36ad4ba28"
format_stamp: "Formatted at 2026-06-19 14:46:55 on dist-test-slave-2sb1"
server_key: "f3d6e57d17f2dbbb8021ab6193f43d76"
server_key_iv: "af3c2fab706c0f0be52eeed9fafcffd0"
server_key_version: "encryptionkey@0"
I20260619 14:46:55.256470 6035 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880383132139-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880383132139-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880383132139-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:46:55.271376 6035 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:46:55.275502 6035 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:46:55.275756 6035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:46:55.276455 6035 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:46:55.277486 6035 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:46:55.277567 6035 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:55.277642 6035 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:46:55.277690 6035 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:55.290820 6035 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34045
I20260619 14:46:55.290884 6163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34045 every 8 connection(s)
I20260619 14:46:55.291925 6035 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880383132139-392-0/minicluster-data/ts-0/data/info.pb
I20260619 14:46:55.300114 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 6035
I20260619 14:46:55.300294 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880383132139-392-0/minicluster-data/ts-0/wal/instance
I20260619 14:46:55.300638 392 external_mini_cluster.cc:1428] Setting key d9fccf573dd8f191aa0b814bb9de175c
Jun 19 14:46:55 dist-test-slave-2sb1 krb5kdc[5928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880415, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:46:55.310158 6012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:55.294233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53553 (local address 127.0.98.62:43381)
0619 14:46:55.294378 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:55.294382 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:55.295505 (+ 1123us) server_negotiation.cc:407] Connection header received
0619 14:46:55.297125 (+ 1620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:55.297128 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:55.297191 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:55.297275 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:55.298566 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:55.299104 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:55.300532 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:55.300900 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:55.301037 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:55.304610 (+ 3573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:55.304620 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:55.304623 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:55.304760 (+ 137us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:55.306770 (+ 2010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:55.307670 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:55.307674 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:55.307676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:55.307784 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:55.308471 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:55.308475 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:55.308477 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:55.308819 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:55.308918 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:55.309830 (+ 912us) server_negotiation.cc:299] Negotiation successful
0619 14:46:55.310017 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 14:46:55.311698 6166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:55.294697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43381 (local address 127.0.98.1:53553)
0619 14:46:55.295282 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:55.295331 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:55.296871 (+ 1540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:55.297457 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:55.297465 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:55.297863 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:46:55.298383 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:55.298398 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:55.299251 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:55.299260 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:55.300225 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:55.300235 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:55.301165 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:55.301170 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:55.301354 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:55.302048 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:46:55.302092 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:46:55.304093 (+ 2001us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:46:55.306947 (+ 2854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:55.306959 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:55.306982 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:55.307511 (+ 529us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:55.307917 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:55.307922 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:55.307926 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:55.308311 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:55.309039 (+ 728us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:46:55.309048 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:46:55.309495 (+ 447us) client_negotiation.cc:769] Sending connection context
0619 14:46:55.309749 (+ 254us) client_negotiation.cc:240] Negotiation successful
0619 14:46:55.310295 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":132,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 14:46:55.313381 6164 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43381
I20260619 14:46:55.313678 6164 heartbeater.cc:464] Registering TS with master...
I20260619 14:46:55.314342 6164 heartbeater.cc:511] Master 127.0.98.62:43381 requested a full tablet report, sending...
I20260619 14:46:55.316224 5973 ts_manager.cc:194] Registered new tserver with Master: 5c29e6d0c91e4a27abf53fe36ad4ba28 (127.0.98.1:34045)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 14:46:55.318172 5973 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:53553
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880383132139-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880383132139-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 14:46:55.338547 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880383132139-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880383132139-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880383132139-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880383132139-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43381
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880383132139-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:46:55.460556 6169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:46:55.460961 6169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:46:55.461068 6169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:46:55.465586 6169 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:46:55.465708 6169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:46:55.465816 6169 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 14:46:55.471611 6169 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880383132139-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880383132139-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880383132139-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880383132139-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43381
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6169
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880383132139-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:46:55.472863 6169 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:46:55.473927 6169 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:46:55.484495 6175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:55.484510 6177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:55.484501 6174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:55.484562 6169 server_base.cc:1034] running on GCE node
I20260619 14:46:55.485159 6169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:46:55.485981 6169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:46:55.487275 6169 hybrid_clock.cc:648] HybridClock initialized: now 1781880415487239 us; error 45 us; skew 500 ppm
Jun 19 14:46:55 dist-test-slave-2sb1 krb5kdc[5928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880415, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:46:55.490792 6169 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 14:46:55.491979 6169 webserver.cc:460] Webserver started at http://127.0.98.2:42277/ using document root <none> and password file <none>
I20260619 14:46:55.492766 6169 fs_manager.cc:359] Metadata directory not provided
I20260619 14:46:55.492914 6169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:46:55.493158 6169 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:46:55.495425 6169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880383132139-392-0/minicluster-data/ts-1/data/instance:
uuid: "3f1db032d7064e9ead60d7d877e4a914"
format_stamp: "Formatted at 2026-06-19 14:46:55 on dist-test-slave-2sb1"
server_key: "201985a5b82343928d75cf770bf00d75"
server_key_iv: "5dfaf7e15dcf1de23a306770ca1f6f52"
server_key_version: "encryptionkey@0"
I20260619 14:46:55.496055 6169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880383132139-392-0/minicluster-data/ts-1/wal/instance:
uuid: "3f1db032d7064e9ead60d7d877e4a914"
format_stamp: "Formatted at 2026-06-19 14:46:55 on dist-test-slave-2sb1"
server_key: "201985a5b82343928d75cf770bf00d75"
server_key_iv: "5dfaf7e15dcf1de23a306770ca1f6f52"
server_key_version: "encryptionkey@0"
I20260619 14:46:55.500963 6169 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260619 14:46:55.503944 6184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:55.505553 6169 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:46:55.505730 6169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880383132139-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880383132139-392-0/minicluster-data/ts-1/wal
uuid: "3f1db032d7064e9ead60d7d877e4a914"
format_stamp: "Formatted at 2026-06-19 14:46:55 on dist-test-slave-2sb1"
server_key: "201985a5b82343928d75cf770bf00d75"
server_key_iv: "5dfaf7e15dcf1de23a306770ca1f6f52"
server_key_version: "encryptionkey@0"
I20260619 14:46:55.505875 6169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880383132139-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880383132139-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880383132139-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:46:55.525805 6169 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:46:55.530068 6169 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:46:55.530315 6169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:46:55.530978 6169 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:46:55.531904 6169 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:46:55.531984 6169 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:55.532059 6169 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:46:55.532114 6169 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:55.543036 6169 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43287
I20260619 14:46:55.543052 6297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43287 every 8 connection(s)
I20260619 14:46:55.544098 6169 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880383132139-392-0/minicluster-data/ts-1/data/info.pb
I20260619 14:46:55.546761 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 6169
I20260619 14:46:55.546885 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880383132139-392-0/minicluster-data/ts-1/wal/instance
I20260619 14:46:55.547159 392 external_mini_cluster.cc:1428] Setting key 0a33af8f920969b8a75fe55d21da275f
Jun 19 14:46:55 dist-test-slave-2sb1 krb5kdc[5928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880415, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260619 14:46:55.560724 6012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:55.546348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41899 (local address 127.0.98.62:43381)
0619 14:46:55.546622 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:55.546627 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:55.547616 (+ 989us) server_negotiation.cc:407] Connection header received
0619 14:46:55.549125 (+ 1509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:55.549131 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:55.549193 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:55.549290 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:55.550740 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:55.551358 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:55.552682 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:55.552941 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:55.553067 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:55.556218 (+ 3151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:55.556222 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:55.556225 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:55.556254 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:55.558114 (+ 1860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:55.558817 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:55.558821 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:55.558823 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:55.558946 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:55.559339 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:55.559343 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:55.559345 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:55.559600 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:55.559716 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:55.560347 (+ 631us) server_negotiation.cc:299] Negotiation successful
0619 14:46:55.560495 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 14:46:55.562012 6300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:55.546906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43381 (local address 127.0.98.2:41899)
0619 14:46:55.547430 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:55.547485 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:55.548876 (+ 1391us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:55.549607 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:55.549616 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:55.550025 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:46:55.550549 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:55.550565 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:55.551545 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:55.551548 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:55.552517 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:55.552527 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:55.553118 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:55.553122 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:55.553309 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:55.554069 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:46:55.554089 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:46:55.555966 (+ 1877us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:46:55.558300 (+ 2334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:55.558308 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:55.558330 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:55.558695 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:55.559058 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:55.559062 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:55.559064 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:55.559237 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:55.559717 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:46:55.559725 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:46:55.560085 (+ 360us) client_negotiation.cc:769] Sending connection context
0619 14:46:55.560307 (+ 222us) client_negotiation.cc:240] Negotiation successful
0619 14:46:55.560715 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":159,"threads_started":1}
I20260619 14:46:55.563633 6298 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43381
I20260619 14:46:55.563921 6298 heartbeater.cc:464] Registering TS with master...
I20260619 14:46:55.564615 6298 heartbeater.cc:511] Master 127.0.98.62:43381 requested a full tablet report, sending...
I20260619 14:46:55.566306 5973 ts_manager.cc:194] Registered new tserver with Master: 3f1db032d7064e9ead60d7d877e4a914 (127.0.98.2:43287)
I20260619 14:46:55.567201 5973 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41899
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880383132139-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880383132139-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 14:46:55.585865 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880383132139-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880383132139-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880383132139-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880383132139-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43381
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880383132139-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:46:55.717689 6303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:46:55.717993 6303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:46:55.718040 6303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:46:55.722510 6303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:46:55.722606 6303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:46:55.722847 6303 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 14:46:55.728514 6303 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880383132139-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880383132139-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880383132139-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880383132139-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43381
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6303
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880383132139-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:46:55.730012 6303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:46:55.731189 6303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:46:55.742102 6309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:55.742058 6308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:55.742076 6311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:55.742424 6303 server_base.cc:1034] running on GCE node
I20260619 14:46:55.742965 6303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:46:55.743783 6303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:46:55.744971 6303 hybrid_clock.cc:648] HybridClock initialized: now 1781880415744951 us; error 33 us; skew 500 ppm
Jun 19 14:46:55 dist-test-slave-2sb1 krb5kdc[5928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880415, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:46:55.748198 6303 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 14:46:55.749297 6303 webserver.cc:460] Webserver started at http://127.0.98.3:42679/ using document root <none> and password file <none>
I20260619 14:46:55.749861 6303 fs_manager.cc:359] Metadata directory not provided
I20260619 14:46:55.749926 6303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:46:55.750100 6303 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:46:55.753396 6303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880383132139-392-0/minicluster-data/ts-2/data/instance:
uuid: "fbdd4c137e4b43fa8cc5114cf3361b25"
format_stamp: "Formatted at 2026-06-19 14:46:55 on dist-test-slave-2sb1"
server_key: "388915336472d22490aa18fc68232af0"
server_key_iv: "125756594edad204e2994c5fd884beac"
server_key_version: "encryptionkey@0"
I20260619 14:46:55.754129 6303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880383132139-392-0/minicluster-data/ts-2/wal/instance:
uuid: "fbdd4c137e4b43fa8cc5114cf3361b25"
format_stamp: "Formatted at 2026-06-19 14:46:55 on dist-test-slave-2sb1"
server_key: "388915336472d22490aa18fc68232af0"
server_key_iv: "125756594edad204e2994c5fd884beac"
server_key_version: "encryptionkey@0"
I20260619 14:46:55.758805 6303 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 14:46:55.762301 6318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:55.764118 6303 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260619 14:46:55.764318 6303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880383132139-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880383132139-392-0/minicluster-data/ts-2/wal
uuid: "fbdd4c137e4b43fa8cc5114cf3361b25"
format_stamp: "Formatted at 2026-06-19 14:46:55 on dist-test-slave-2sb1"
server_key: "388915336472d22490aa18fc68232af0"
server_key_iv: "125756594edad204e2994c5fd884beac"
server_key_version: "encryptionkey@0"
I20260619 14:46:55.764482 6303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880383132139-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880383132139-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880383132139-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:46:55.783097 6303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:46:55.787084 6303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:46:55.787369 6303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:46:55.788340 6303 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:46:55.789288 6303 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:46:55.789371 6303 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:55.789444 6303 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:46:55.789510 6303 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:55.800936 6303 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41837
I20260619 14:46:55.801034 6431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41837 every 8 connection(s)
I20260619 14:46:55.802033 6303 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880383132139-392-0/minicluster-data/ts-2/data/info.pb
I20260619 14:46:55.804979 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 6303
I20260619 14:46:55.805096 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880383132139-392-0/minicluster-data/ts-2/wal/instance
I20260619 14:46:55.805506 392 external_mini_cluster.cc:1428] Setting key 12a33f194e58f80eba8032d6420900da
Jun 19 14:46:55 dist-test-slave-2sb1 krb5kdc[5928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880415, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:46:55.819207 6012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:55.804497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42531 (local address 127.0.98.62:43381)
0619 14:46:55.804806 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:55.804812 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:55.805881 (+ 1069us) server_negotiation.cc:407] Connection header received
0619 14:46:55.807462 (+ 1581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:55.807466 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:55.807523 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:55.807607 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:55.809002 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:55.809541 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:55.810817 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:55.811130 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:55.811246 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:55.814286 (+ 3040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:55.814289 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:55.814292 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:55.814320 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:55.816309 (+ 1989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:55.817250 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:55.817253 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:55.817255 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:55.817349 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:55.817816 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:55.817819 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:55.817820 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:55.818077 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:55.818220 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:55.818869 (+ 649us) server_negotiation.cc:299] Negotiation successful
0619 14:46:55.819048 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153}
I20260619 14:46:55.820180 6434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:55.804980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43381 (local address 127.0.98.3:42531)
0619 14:46:55.805682 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:55.805750 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:55.807227 (+ 1477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:55.807835 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:55.807847 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:55.808275 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:46:55.808850 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:55.808866 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:55.809682 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:55.809691 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:55.810680 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:55.810690 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:55.811288 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:55.811292 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:55.811473 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:55.812114 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:46:55.812132 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:46:55.814002 (+ 1870us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:46:55.816587 (+ 2585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:55.816595 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:55.816614 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:55.817060 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:55.817548 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:55.817550 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:55.817552 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:55.817712 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:55.818217 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:46:55.818225 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:46:55.818542 (+ 317us) client_negotiation.cc:769] Sending connection context
0619 14:46:55.818764 (+ 222us) client_negotiation.cc:240] Negotiation successful
0619 14:46:55.819150 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":182,"threads_started":1}
I20260619 14:46:55.821731 6432 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43381
I20260619 14:46:55.822161 6432 heartbeater.cc:464] Registering TS with master...
I20260619 14:46:55.822803 6432 heartbeater.cc:511] Master 127.0.98.62:43381 requested a full tablet report, sending...
I20260619 14:46:55.824050 5973 ts_manager.cc:194] Registered new tserver with Master: fbdd4c137e4b43fa8cc5114cf3361b25 (127.0.98.3:41837)
I20260619 14:46:55.824715 5973 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42531
I20260619 14:46:55.833266 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880383132139-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880383132139-392-0/krb5kdc/test-admin.keytab.
Jun 19 14:46:55 dist-test-slave-2sb1 krb5kdc[5928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880415, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:46:55.855647 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260619 14:46:55.859314 6439 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:46:55.857592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37846 (local address 127.0.98.1:34045)
0619 14:46:55.857947 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:55.857954 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:55.857987 (+ 33us) server_negotiation.cc:407] Connection header received
0619 14:46:55.858036 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:55.858041 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:55.858224 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:55.858349 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:55.859144 (+ 795us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37846: BlockingRecv error: recv got EOF from 127.0.0.1:37846 (error 108)
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":145,"threads_started":1}
W20260619 14:46:55.859841 392 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880383132139-392-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880383132139-392-0/krb5kdc/krb5cc)
Jun 19 14:46:55 dist-test-slave-2sb1 krb5kdc[5928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880415, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:46:55.861159 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 14:46:55 dist-test-slave-2sb1 krb5kdc[5928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880415, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:46:55.862202 392 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 14:46:55 dist-test-slave-2sb1 krb5kdc[5928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880415, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 14:46:55.873871 6439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:55.863371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37860 (local address 127.0.98.1:34045)
0619 14:46:55.863534 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:55.863539 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:55.863729 (+ 190us) server_negotiation.cc:407] Connection header received
0619 14:46:55.863901 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:55.863904 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:55.863949 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:55.864022 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:55.864639 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:55.865432 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:55.866302 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:55.866718 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:55.866840 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:55.869559 (+ 2719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:55.869571 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:55.869585 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:55.869615 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:55.871618 (+ 2003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:55.872218 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:55.872224 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:55.872225 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:55.872316 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:55.872697 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:55.872701 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:55.872703 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:55.872983 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:55.873240 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:55.873403 (+ 163us) server_negotiation.cc:299] Negotiation successful
0619 14:46:55.873741 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260619 14:46:55.879225 6439 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:46:55.878010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37862 (local address 127.0.98.1:34045)
0619 14:46:55.878194 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:55.878198 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:55.878214 (+ 16us) server_negotiation.cc:407] Connection header received
0619 14:46:55.878336 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:55.878339 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:55.878385 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:55.878466 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:55.879067 (+ 601us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37862: BlockingRecv error: recv got EOF from 127.0.0.1:37862 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260619 14:46:55.879233 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880383132139-392-0/krb5kdc/krb5cc' of type FILE
Jun 19 14:46:55 dist-test-slave-2sb1 krb5kdc[5928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880415, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:46:55.880861 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 14:46:55 dist-test-slave-2sb1 krb5kdc[5928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880415, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:46:55.882368 392 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 14:46:55 dist-test-slave-2sb1 krb5kdc[5928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880415, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 14:46:55.894589 6439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:55.884205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37878 (local address 127.0.98.1:34045)
0619 14:46:55.884399 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:55.884402 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:55.884419 (+ 17us) server_negotiation.cc:407] Connection header received
0619 14:46:55.884559 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:55.884562 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:55.884603 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:55.884764 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:55.885545 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:55.886055 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:55.886980 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:55.887230 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:55.887328 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:55.890651 (+ 3323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:55.890659 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:55.890665 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:55.890714 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:55.892744 (+ 2030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:55.893251 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:55.893255 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:55.893257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:55.893354 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:55.893637 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:55.893640 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:55.893641 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:55.893834 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:55.893992 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:55.894145 (+ 153us) server_negotiation.cc:299] Negotiation successful
0619 14:46:55.894249 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
W20260619 14:46:55.897897 6439 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:46:55.896698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37890 (local address 127.0.98.1:34045)
0619 14:46:55.896816 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:55.896819 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:55.896857 (+ 38us) server_negotiation.cc:407] Connection header received
0619 14:46:55.897075 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:55.897078 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:55.897118 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:55.897190 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:55.897778 (+ 588us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37890: BlockingRecv error: recv got EOF from 127.0.0.1:37890 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260619 14:46:55.897949 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781880383132139-392-0/krb5kdc/krb5cc' of type FILE
Jun 19 14:46:55 dist-test-slave-2sb1 krb5kdc[5928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880415, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:46:55.899089 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 19 14:46:55 dist-test-slave-2sb1 krb5kdc[5928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880415, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:46:55.900166 392 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 19 14:46:55 dist-test-slave-2sb1 krb5kdc[5928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880415, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 14:46:55.911325 6439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:55.901305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37900 (local address 127.0.98.1:34045)
0619 14:46:55.901434 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:55.901437 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:55.901615 (+ 178us) server_negotiation.cc:407] Connection header received
0619 14:46:55.901659 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:55.901661 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:55.901703 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:55.901779 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:55.902440 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:55.903067 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:55.904026 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:55.904243 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:55.904328 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:55.907325 (+ 2997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:55.907329 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:55.907331 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:55.907362 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:55.909234 (+ 1872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:55.909821 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:55.909825 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:55.909826 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:55.909917 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:55.910365 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:55.910368 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:55.910369 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:55.910575 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:55.910715 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:55.910980 (+ 265us) server_negotiation.cc:299] Negotiation successful
0619 14:46:55.911089 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 14:46:55.912789 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 6035
I20260619 14:46:55.920192 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 6169
I20260619 14:46:55.927242 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 6303
I20260619 14:46:55.933557 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 5941
2026-06-19T14:46:55Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2437 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:46:56 dist-test-slave-2sb1 krb5kdc[6452](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:46:56 dist-test-slave-2sb1 krb5kdc[6452](info): set up 2 sockets
Jun 19 14:46:56 dist-test-slave-2sb1 krb5kdc[6452](info): commencing operation
krb5kdc: starting...
W20260619 14:46:57.042420 392 mini_kdc.cc:121] Time spent starting KDC: real 1.086s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/krb5kdc/test-user.keytab.
Jun 19 14:46:57 dist-test-slave-2sb1 krb5kdc[6452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880417, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:46:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:46:57Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260619 14:46:57.193799 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37899
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37899
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:46:57.315305 6465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:46:57.315668 6465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:46:57.315750 6465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:46:57.320982 6465 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:46:57.321082 6465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:46:57.321120 6465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:46:57.321179 6465 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:46:57.321237 6465 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:46:57.327154 6465 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37899
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37899
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6465
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:46:57.328374 6465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:46:57.329560 6465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:46:57.338928 6471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:57.339032 6470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:57.338958 6473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:57.339337 6465 server_base.cc:1034] running on GCE node
I20260619 14:46:57.340099 6465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:46:57.341341 6465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:46:57.342594 6465 hybrid_clock.cc:648] HybridClock initialized: now 1781880417342562 us; error 48 us; skew 500 ppm
Jun 19 14:46:57 dist-test-slave-2sb1 krb5kdc[6452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880417, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:46:57.346642 6465 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260619 14:46:57.347733 6465 webserver.cc:460] Webserver started at http://127.0.98.62:40247/ using document root <none> and password file <none>
I20260619 14:46:57.348383 6465 fs_manager.cc:359] Metadata directory not provided
I20260619 14:46:57.348459 6465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:46:57.348692 6465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:46:57.351495 6465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/master-0/data/instance:
uuid: "4a2a5f1050da4c09b1351fb6ecb3986a"
format_stamp: "Formatted at 2026-06-19 14:46:57 on dist-test-slave-2sb1"
server_key: "024c1f47724e4ecedbc77030ea6b0994"
server_key_iv: "b290c23b04cf8d7a238aca07987aa667"
server_key_version: "encryptionkey@0"
I20260619 14:46:57.352345 6465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/master-0/wal/instance:
uuid: "4a2a5f1050da4c09b1351fb6ecb3986a"
format_stamp: "Formatted at 2026-06-19 14:46:57 on dist-test-slave-2sb1"
server_key: "024c1f47724e4ecedbc77030ea6b0994"
server_key_iv: "b290c23b04cf8d7a238aca07987aa667"
server_key_version: "encryptionkey@0"
I20260619 14:46:57.356829 6465 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.001s
I20260619 14:46:57.359809 6480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:57.361348 6465 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:46:57.361532 6465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/master-0/wal
uuid: "4a2a5f1050da4c09b1351fb6ecb3986a"
format_stamp: "Formatted at 2026-06-19 14:46:57 on dist-test-slave-2sb1"
server_key: "024c1f47724e4ecedbc77030ea6b0994"
server_key_iv: "b290c23b04cf8d7a238aca07987aa667"
server_key_version: "encryptionkey@0"
I20260619 14:46:57.361670 6465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:46:57.381740 6465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:46:57.385414 6465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:46:57.385666 6465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:46:57.393359 6465 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37899
I20260619 14:46:57.393785 6532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37899 every 8 connection(s)
I20260619 14:46:57.394481 6465 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:46:57.397595 6533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:46:57.401535 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 6465
I20260619 14:46:57.401678 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/master-0/wal/instance
I20260619 14:46:57.402024 392 external_mini_cluster.cc:1428] Setting key 2866356d586464e4f1ed5a1ac04123be
I20260619 14:46:57.404065 6533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a2a5f1050da4c09b1351fb6ecb3986a: Bootstrap starting.
I20260619 14:46:57.406383 6533 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a2a5f1050da4c09b1351fb6ecb3986a: Neither blocks nor log segments found. Creating new log.
I20260619 14:46:57.407176 6533 log.cc:824] T 00000000000000000000000000000000 P 4a2a5f1050da4c09b1351fb6ecb3986a: Log is configured to *not* fsync() on all Append() calls
I20260619 14:46:57.409571 6533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a2a5f1050da4c09b1351fb6ecb3986a: No bootstrap required, opened a new log
Jun 19 14:46:57 dist-test-slave-2sb1 krb5kdc[6452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880417, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 14:46:57.414443 6533 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a2a5f1050da4c09b1351fb6ecb3986a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a2a5f1050da4c09b1351fb6ecb3986a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37899 } }
I20260619 14:46:57.414743 6533 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4a2a5f1050da4c09b1351fb6ecb3986a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:46:57.414814 6533 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a2a5f1050da4c09b1351fb6ecb3986a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a2a5f1050da4c09b1351fb6ecb3986a, State: Initialized, Role: FOLLOWER
I20260619 14:46:57.415407 6533 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a2a5f1050da4c09b1351fb6ecb3986a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a2a5f1050da4c09b1351fb6ecb3986a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37899 } }
I20260619 14:46:57.415546 6533 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4a2a5f1050da4c09b1351fb6ecb3986a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:46:57.415638 6533 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a2a5f1050da4c09b1351fb6ecb3986a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:46:57.415740 6533 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a2a5f1050da4c09b1351fb6ecb3986a [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:46:57.417131 6533 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a2a5f1050da4c09b1351fb6ecb3986a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a2a5f1050da4c09b1351fb6ecb3986a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37899 } }
I20260619 14:46:57.417575 6533 leader_election.cc:302] T 00000000000000000000000000000000 P 4a2a5f1050da4c09b1351fb6ecb3986a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a2a5f1050da4c09b1351fb6ecb3986a; no voters:
I20260619 14:46:57.417912 6533 leader_election.cc:288] T 00000000000000000000000000000000 P 4a2a5f1050da4c09b1351fb6ecb3986a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:46:57.418105 6538 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a2a5f1050da4c09b1351fb6ecb3986a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:46:57.418342 6538 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a2a5f1050da4c09b1351fb6ecb3986a [term 1 LEADER]: Becoming Leader. State: Replica: 4a2a5f1050da4c09b1351fb6ecb3986a, State: Running, Role: LEADER
I20260619 14:46:57.418759 6538 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a2a5f1050da4c09b1351fb6ecb3986a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a2a5f1050da4c09b1351fb6ecb3986a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37899 } }
I20260619 14:46:57.419644 6533 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a2a5f1050da4c09b1351fb6ecb3986a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:46:57.420264 6538 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a2a5f1050da4c09b1351fb6ecb3986a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a2a5f1050da4c09b1351fb6ecb3986a. Latest consensus state: current_term: 1 leader_uuid: "4a2a5f1050da4c09b1351fb6ecb3986a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a2a5f1050da4c09b1351fb6ecb3986a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37899 } } }
I20260619 14:46:57.420405 6538 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a2a5f1050da4c09b1351fb6ecb3986a [sys.catalog]: This master's current role is: LEADER
I20260619 14:46:57.421185 6536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:57.403625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33474 (local address 127.0.98.62:37899)
0619 14:46:57.404168 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:57.404195 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:57.404242 (+ 47us) server_negotiation.cc:407] Connection header received
0619 14:46:57.405005 (+ 763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:57.405024 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:57.405358 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:57.405697 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:57.406558 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:57.407680 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:57.409125 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:57.409759 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:57.409921 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:57.413319 (+ 3398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:57.413326 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:57.413341 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:57.413382 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:57.415895 (+ 2513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:57.416718 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:57.416726 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:57.416732 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:57.416873 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:57.417316 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:57.417322 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:57.417325 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:57.418127 (+ 802us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:57.419406 (+ 1279us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:57.419584 (+ 178us) server_negotiation.cc:299] Negotiation successful
0619 14:46:57.419899 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":132,"threads_started":1}
I20260619 14:46:57.421408 6539 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a2a5f1050da4c09b1351fb6ecb3986a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4a2a5f1050da4c09b1351fb6ecb3986a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a2a5f1050da4c09b1351fb6ecb3986a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37899 } } }
I20260619 14:46:57.421512 6539 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a2a5f1050da4c09b1351fb6ecb3986a [sys.catalog]: This master's current role is: LEADER
W20260619 14:46:57.426378 6554 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4a2a5f1050da4c09b1351fb6ecb3986a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:46:57.426501 6554 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:46:57.426589 6542 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:46:57.427309 6542 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:46:57.433243 6542 catalog_manager.cc:1347] Generated new cluster ID: 54bc711fead8420ebecba113ea139541
I20260619 14:46:57.433359 6542 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:46:57.448912 6542 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:46:57.450120 6542 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:46:57.454571 6542 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4a2a5f1050da4c09b1351fb6ecb3986a: Generated new TSK 0
I20260619 14:46:57.455345 6542 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260619 14:46:57.496960 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37899
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:46:57.616959 6559 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:46:57.617316 6559 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:46:57.617372 6559 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:46:57.621933 6559 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:46:57.622017 6559 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:46:57.622088 6559 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 14:46:57.627995 6559 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37899
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6559
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:46:57.629266 6559 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:46:57.630316 6559 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:46:57.641546 6564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:57.641532 6567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:57.641592 6565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:57.641898 6559 server_base.cc:1034] running on GCE node
I20260619 14:46:57.642721 6559 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:46:57.643503 6559 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:46:57.644757 6559 hybrid_clock.cc:648] HybridClock initialized: now 1781880417644745 us; error 32 us; skew 500 ppm
Jun 19 14:46:57 dist-test-slave-2sb1 krb5kdc[6452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880417, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:46:57.648573 6559 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260619 14:46:57.649857 6559 webserver.cc:460] Webserver started at http://127.0.98.1:35971/ using document root <none> and password file <none>
I20260619 14:46:57.650502 6559 fs_manager.cc:359] Metadata directory not provided
I20260619 14:46:57.650609 6559 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:46:57.650851 6559 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:46:57.653193 6559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/ts-0/data/instance:
uuid: "1b116e7a13764f34b66cdbb5e53338e4"
format_stamp: "Formatted at 2026-06-19 14:46:57 on dist-test-slave-2sb1"
server_key: "896c6fadbbc1d7ac99ea7218c02cc486"
server_key_iv: "1aab366100b84b6a7657b0cd27c053d2"
server_key_version: "encryptionkey@0"
I20260619 14:46:57.653847 6559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/ts-0/wal/instance:
uuid: "1b116e7a13764f34b66cdbb5e53338e4"
format_stamp: "Formatted at 2026-06-19 14:46:57 on dist-test-slave-2sb1"
server_key: "896c6fadbbc1d7ac99ea7218c02cc486"
server_key_iv: "1aab366100b84b6a7657b0cd27c053d2"
server_key_version: "encryptionkey@0"
I20260619 14:46:57.658341 6559 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 14:46:57.661887 6574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:57.663432 6559 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 14:46:57.663630 6559 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/ts-0/wal
uuid: "1b116e7a13764f34b66cdbb5e53338e4"
format_stamp: "Formatted at 2026-06-19 14:46:57 on dist-test-slave-2sb1"
server_key: "896c6fadbbc1d7ac99ea7218c02cc486"
server_key_iv: "1aab366100b84b6a7657b0cd27c053d2"
server_key_version: "encryptionkey@0"
I20260619 14:46:57.663781 6559 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:46:57.692549 6559 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:46:57.696640 6559 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:46:57.696939 6559 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:46:57.697585 6559 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:46:57.698462 6559 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:46:57.698535 6559 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:57.698606 6559 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:46:57.698655 6559 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:57.709671 6559 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37475
I20260619 14:46:57.709709 6687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37475 every 8 connection(s)
I20260619 14:46:57.710669 6559 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/ts-0/data/info.pb
I20260619 14:46:57.714599 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 6559
I20260619 14:46:57.714697 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/ts-0/wal/instance
I20260619 14:46:57.714975 392 external_mini_cluster.cc:1428] Setting key a346458791ebfd86b3c05832ea06eeac
Jun 19 14:46:57 dist-test-slave-2sb1 krb5kdc[6452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880417, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 14:46:57.727676 6536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:57.712878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51779 (local address 127.0.98.62:37899)
0619 14:46:57.713033 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:57.713039 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:57.714108 (+ 1069us) server_negotiation.cc:407] Connection header received
0619 14:46:57.715657 (+ 1549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:57.715660 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:57.715713 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:57.715804 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:57.717280 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:57.718165 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:57.719697 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:57.720069 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:57.720207 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:57.723518 (+ 3311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:57.723523 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:57.723526 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:57.723559 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:57.725280 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:57.725917 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:57.725922 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:57.725923 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:57.726006 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:57.726394 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:57.726398 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:57.726400 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:57.726617 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:57.726748 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:57.727321 (+ 573us) server_negotiation.cc:299] Negotiation successful
0619 14:46:57.727481 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
I20260619 14:46:57.728796 6690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:57.713368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37899 (local address 127.0.98.1:51779)
0619 14:46:57.713870 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:57.713926 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:57.715442 (+ 1516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:57.716079 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:57.716089 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:57.716550 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:46:57.717121 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:57.717135 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:57.718361 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:57.718379 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:57.719498 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:57.719509 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:57.720269 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:57.720273 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:57.720475 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:57.721370 (+ 895us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:46:57.721417 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:46:57.723268 (+ 1851us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:46:57.725434 (+ 2166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:57.725442 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:57.725464 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:57.725783 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:57.726112 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:57.726116 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:57.726118 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:57.726283 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:57.726748 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:46:57.726757 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:46:57.727057 (+ 300us) client_negotiation.cc:769] Sending connection context
0619 14:46:57.727265 (+ 208us) client_negotiation.cc:240] Negotiation successful
0619 14:46:57.727644 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":113,"threads_started":1}
I20260619 14:46:57.730337 6688 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37899
I20260619 14:46:57.730629 6688 heartbeater.cc:464] Registering TS with master...
I20260619 14:46:57.731185 6688 heartbeater.cc:511] Master 127.0.98.62:37899 requested a full tablet report, sending...
I20260619 14:46:57.733013 6497 ts_manager.cc:194] Registered new tserver with Master: 1b116e7a13764f34b66cdbb5e53338e4 (127.0.98.1:37475)
I20260619 14:46:57.734292 6497 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:51779
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260619 14:46:57.749406 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37899
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:46:57.865269 6693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:46:57.865602 6693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:46:57.865675 6693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:46:57.870294 6693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:46:57.870378 6693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:46:57.870478 6693 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 14:46:57.876708 6693 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37899
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6693
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:46:57.877884 6693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:46:57.878957 6693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:46:57.890050 6701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:57.890007 6698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:57.890020 6699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:57.890286 6693 server_base.cc:1034] running on GCE node
I20260619 14:46:57.890935 6693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:46:57.891741 6693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:46:57.892983 6693 hybrid_clock.cc:648] HybridClock initialized: now 1781880417892946 us; error 68 us; skew 500 ppm
Jun 19 14:46:57 dist-test-slave-2sb1 krb5kdc[6452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880417, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:46:57.896414 6693 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260619 14:46:57.897487 6693 webserver.cc:460] Webserver started at http://127.0.98.2:44951/ using document root <none> and password file <none>
I20260619 14:46:57.898105 6693 fs_manager.cc:359] Metadata directory not provided
I20260619 14:46:57.898203 6693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:46:57.898445 6693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:46:57.900556 6693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/ts-1/data/instance:
uuid: "0f513b290a8c49ee81a89fa8d2152c04"
format_stamp: "Formatted at 2026-06-19 14:46:57 on dist-test-slave-2sb1"
server_key: "337f40618b3e9f51a1c1e87bd723c09c"
server_key_iv: "5cc90915ec433651079b3418b4af4b63"
server_key_version: "encryptionkey@0"
I20260619 14:46:57.901258 6693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/ts-1/wal/instance:
uuid: "0f513b290a8c49ee81a89fa8d2152c04"
format_stamp: "Formatted at 2026-06-19 14:46:57 on dist-test-slave-2sb1"
server_key: "337f40618b3e9f51a1c1e87bd723c09c"
server_key_iv: "5cc90915ec433651079b3418b4af4b63"
server_key_version: "encryptionkey@0"
I20260619 14:46:57.905455 6693 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 14:46:57.908130 6708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:57.909672 6693 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 14:46:57.909835 6693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/ts-1/wal
uuid: "0f513b290a8c49ee81a89fa8d2152c04"
format_stamp: "Formatted at 2026-06-19 14:46:57 on dist-test-slave-2sb1"
server_key: "337f40618b3e9f51a1c1e87bd723c09c"
server_key_iv: "5cc90915ec433651079b3418b4af4b63"
server_key_version: "encryptionkey@0"
I20260619 14:46:57.909965 6693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:46:57.924090 6693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:46:57.927562 6693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:46:57.927793 6693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:46:57.928402 6693 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:46:57.929335 6693 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:46:57.929411 6693 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:57.929483 6693 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:46:57.929536 6693 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:57.939601 6693 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37385
I20260619 14:46:57.939628 6821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37385 every 8 connection(s)
I20260619 14:46:57.940541 6693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/ts-1/data/info.pb
I20260619 14:46:57.946794 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 6693
I20260619 14:46:57.946941 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/ts-1/wal/instance
I20260619 14:46:57.947238 392 external_mini_cluster.cc:1428] Setting key 19556a4ba114b57b8bebc251fd09eab6
Jun 19 14:46:57 dist-test-slave-2sb1 krb5kdc[6452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880417, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 14:46:57.960084 6536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:57.942912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54185 (local address 127.0.98.62:37899)
0619 14:46:57.943087 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:57.943092 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:57.944011 (+ 919us) server_negotiation.cc:407] Connection header received
0619 14:46:57.945768 (+ 1757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:57.945772 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:57.945840 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:57.945971 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:57.947723 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:57.948426 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:57.950944 (+ 2518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:57.951452 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:57.951588 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:57.954775 (+ 3187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:57.954784 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:57.954790 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:57.954827 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:57.956791 (+ 1964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:57.957775 (+ 984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:57.957782 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:57.957786 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:57.957889 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:57.958438 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:57.958444 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:57.958447 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:57.958700 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:57.958785 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:57.959771 (+ 986us) server_negotiation.cc:299] Negotiation successful
0619 14:46:57.959942 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
I20260619 14:46:57.962196 6824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:57.943302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37899 (local address 127.0.98.2:54185)
0619 14:46:57.943800 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:57.943846 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:57.945514 (+ 1668us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:57.946168 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:57.946175 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:57.946712 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:46:57.947498 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:57.947522 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:57.948691 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:57.948698 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:57.950478 (+ 1780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:57.950495 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:57.951665 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:57.951669 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:57.951848 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:57.952587 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:46:57.952607 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:46:57.954537 (+ 1930us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:46:57.957046 (+ 2509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:57.957054 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:57.957079 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:57.957601 (+ 522us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:57.958059 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:57.958062 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:57.958063 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:57.958297 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:57.958822 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:46:57.958828 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:46:57.959115 (+ 287us) client_negotiation.cc:769] Sending connection context
0619 14:46:57.959298 (+ 183us) client_negotiation.cc:240] Negotiation successful
0619 14:46:57.959834 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":106,"threads_started":1}
I20260619 14:46:57.964031 6822 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37899
I20260619 14:46:57.964365 6822 heartbeater.cc:464] Registering TS with master...
I20260619 14:46:57.965174 6822 heartbeater.cc:511] Master 127.0.98.62:37899 requested a full tablet report, sending...
I20260619 14:46:57.966566 6497 ts_manager.cc:194] Registered new tserver with Master: 0f513b290a8c49ee81a89fa8d2152c04 (127.0.98.2:37385)
I20260619 14:46:57.967453 6497 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:54185
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260619 14:46:57.985246 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37899
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:46:58.105114 6827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:46:58.105413 6827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:46:58.105466 6827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:46:58.109797 6827 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:46:58.109882 6827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:46:58.109946 6827 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 14:46:58.116036 6827 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37899
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6827
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:46:58.117264 6827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:46:58.118340 6827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:46:58.129696 6835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:58.129707 6832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:58.129707 6833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:58.130342 6827 server_base.cc:1034] running on GCE node
I20260619 14:46:58.131023 6827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:46:58.131875 6827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:46:58.133247 6827 hybrid_clock.cc:648] HybridClock initialized: now 1781880418133209 us; error 52 us; skew 500 ppm
Jun 19 14:46:58 dist-test-slave-2sb1 krb5kdc[6452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880418, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:46:58.136507 6827 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260619 14:46:58.137853 6827 webserver.cc:460] Webserver started at http://127.0.98.3:35361/ using document root <none> and password file <none>
I20260619 14:46:58.138504 6827 fs_manager.cc:359] Metadata directory not provided
I20260619 14:46:58.138574 6827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:46:58.138846 6827 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:46:58.141281 6827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/ts-2/data/instance:
uuid: "2cb8830304e145db8591b8ff765eeef3"
format_stamp: "Formatted at 2026-06-19 14:46:58 on dist-test-slave-2sb1"
server_key: "158f7d962baf155821fe6fd4f3400e36"
server_key_iv: "8ef0555896dbc59b5aaba4d9047079f7"
server_key_version: "encryptionkey@0"
I20260619 14:46:58.141950 6827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/ts-2/wal/instance:
uuid: "2cb8830304e145db8591b8ff765eeef3"
format_stamp: "Formatted at 2026-06-19 14:46:58 on dist-test-slave-2sb1"
server_key: "158f7d962baf155821fe6fd4f3400e36"
server_key_iv: "8ef0555896dbc59b5aaba4d9047079f7"
server_key_version: "encryptionkey@0"
I20260619 14:46:58.146536 6827 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 14:46:58.150245 6842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:58.151702 6827 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 14:46:58.151878 6827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/ts-2/wal
uuid: "2cb8830304e145db8591b8ff765eeef3"
format_stamp: "Formatted at 2026-06-19 14:46:58 on dist-test-slave-2sb1"
server_key: "158f7d962baf155821fe6fd4f3400e36"
server_key_iv: "8ef0555896dbc59b5aaba4d9047079f7"
server_key_version: "encryptionkey@0"
I20260619 14:46:58.152032 6827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:46:58.163753 6827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:46:58.167923 6827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:46:58.168146 6827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:46:58.168844 6827 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:46:58.169706 6827 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:46:58.169778 6827 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:58.169849 6827 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:46:58.169901 6827 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:58.179761 6827 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45697
I20260619 14:46:58.179800 6955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45697 every 8 connection(s)
I20260619 14:46:58.180800 6827 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/ts-2/data/info.pb
I20260619 14:46:58.183084 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 6827
I20260619 14:46:58.183212 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/ts-2/wal/instance
I20260619 14:46:58.183705 392 external_mini_cluster.cc:1428] Setting key 3fa557bc01853f720bd445fed96a241c
Jun 19 14:46:58 dist-test-slave-2sb1 krb5kdc[6452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880418, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 14:46:58.196321 6536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:58.182662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39789 (local address 127.0.98.62:37899)
0619 14:46:58.182821 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:58.182824 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:58.183910 (+ 1086us) server_negotiation.cc:407] Connection header received
0619 14:46:58.185669 (+ 1759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:58.185672 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:58.185727 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:58.185805 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:58.187064 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:58.187593 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:58.188824 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:58.189048 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:58.189164 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:58.192142 (+ 2978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:58.192145 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:58.192147 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:58.192174 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:58.193977 (+ 1803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:58.194586 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:58.194590 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:58.194592 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:58.194677 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:58.195050 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:58.195053 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:58.195055 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:58.195245 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:58.195352 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:58.196014 (+ 662us) server_negotiation.cc:299] Negotiation successful
0619 14:46:58.196155 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 14:46:58.197477 6958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:58.183235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37899 (local address 127.0.98.3:39789)
0619 14:46:58.183740 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:58.183783 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:58.185446 (+ 1663us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:58.185987 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:58.185995 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:58.186374 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:46:58.186900 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:58.186914 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:58.187728 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:58.187733 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:58.188691 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:58.188700 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:58.189205 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:58.189209 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:58.189381 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:58.190035 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:46:58.190053 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:46:58.191902 (+ 1849us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:46:58.194120 (+ 2218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:58.194127 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:58.194145 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:58.194442 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:58.194781 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:58.194784 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:58.194786 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:58.194946 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:58.195363 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:46:58.195372 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:46:58.195754 (+ 382us) client_negotiation.cc:769] Sending connection context
0619 14:46:58.195973 (+ 219us) client_negotiation.cc:240] Negotiation successful
0619 14:46:58.196316 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":144,"threads_started":1}
I20260619 14:46:58.199087 6956 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37899
I20260619 14:46:58.199326 6956 heartbeater.cc:464] Registering TS with master...
I20260619 14:46:58.199846 6956 heartbeater.cc:511] Master 127.0.98.62:37899 requested a full tablet report, sending...
I20260619 14:46:58.201081 6497 ts_manager.cc:194] Registered new tserver with Master: 2cb8830304e145db8591b8ff765eeef3 (127.0.98.3:45697)
I20260619 14:46:58.201818 6497 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:39789
I20260619 14:46:58.211206 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260619 14:46:58.244784 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:36857
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37899,127.0.98.61:36857
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:46:58.372723 6961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:46:58.373087 6961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:46:58.373186 6961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:46:58.377683 6961 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:46:58.377797 6961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:46:58.377833 6961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:46:58.377875 6961 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:46:58.377913 6961 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:46:58.383617 6961 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37899,127.0.98.61:36857
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:36857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6961
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
W20260619 14:46:58.383929 6961 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:37899,127.0.98.61:36857'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260619 14:46:58.384935 6961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:46:58.386026 6961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:46:58.395442 6966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:58.395524 6961 server_base.cc:1034] running on GCE node
W20260619 14:46:58.395737 6967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:58.395434 6969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:58.396598 6961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:46:58.397820 6961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:46:58.399051 6961 hybrid_clock.cc:648] HybridClock initialized: now 1781880418399027 us; error 46 us; skew 500 ppm
Jun 19 14:46:58 dist-test-slave-2sb1 krb5kdc[6452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880418, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:46:58.402467 6961 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260619 14:46:58.403576 6961 webserver.cc:460] Webserver started at http://127.0.98.61:33987/ using document root <none> and password file <none>
I20260619 14:46:58.404255 6961 fs_manager.cc:359] Metadata directory not provided
I20260619 14:46:58.404330 6961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:46:58.404533 6961 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:46:58.407226 6961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/master-1/data/instance:
uuid: "8e87b561972740938a4f82c45931705b"
format_stamp: "Formatted at 2026-06-19 14:46:58 on dist-test-slave-2sb1"
server_key: "d865a6b3c25f0d80b8b0e07c39537feb"
server_key_iv: "4f9a5468c1d644355ba10bdbe2ce532d"
server_key_version: "encryptionkey@0"
I20260619 14:46:58.407886 6961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/master-1/wal/instance:
uuid: "8e87b561972740938a4f82c45931705b"
format_stamp: "Formatted at 2026-06-19 14:46:58 on dist-test-slave-2sb1"
server_key: "d865a6b3c25f0d80b8b0e07c39537feb"
server_key_iv: "4f9a5468c1d644355ba10bdbe2ce532d"
server_key_version: "encryptionkey@0"
I20260619 14:46:58.411917 6961 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 14:46:58.414773 6976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:58.416268 6961 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 14:46:58.416469 6961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/master-1/wal
uuid: "8e87b561972740938a4f82c45931705b"
format_stamp: "Formatted at 2026-06-19 14:46:58 on dist-test-slave-2sb1"
server_key: "d865a6b3c25f0d80b8b0e07c39537feb"
server_key_iv: "4f9a5468c1d644355ba10bdbe2ce532d"
server_key_version: "encryptionkey@0"
I20260619 14:46:58.416600 6961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:46:58.461558 6961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:46:58.466111 6961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:46:58.466351 6961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:46:58.475021 6961 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:36857
I20260619 14:46:58.475039 7028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:36857 every 8 connection(s)
I20260619 14:46:58.476143 6961 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/master-1/data/info.pb
I20260619 14:46:58.479521 7029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:46:58.483284 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 6961
I20260619 14:46:58.483952 7029 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37899 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 14:46:58 dist-test-slave-2sb1 krb5kdc[6452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880418, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Jun 19 14:46:58 dist-test-slave-2sb1 krb5kdc[6452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880417, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260619 14:46:58.498157 6536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:58.486357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33476 (local address 127.0.98.62:37899)
0619 14:46:58.486488 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:58.486493 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:58.486664 (+ 171us) server_negotiation.cc:407] Connection header received
0619 14:46:58.487229 (+ 565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:58.487233 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:58.487292 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:58.487403 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:58.488408 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:58.489076 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:58.490250 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:58.490519 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:58.490646 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:58.493970 (+ 3324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:58.493973 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:58.493975 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:58.494002 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:58.495738 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:58.496441 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:58.496445 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:58.496447 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:58.496532 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:58.496925 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:58.496929 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:58.496931 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:58.497164 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:58.497271 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:58.497845 (+ 574us) server_negotiation.cc:299] Negotiation successful
0619 14:46:58.497984 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 14:46:58.499132 7032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:58.485201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42550 (local address 127.0.98.61:36857)
0619 14:46:58.485784 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:58.485798 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:58.485834 (+ 36us) server_negotiation.cc:407] Connection header received
0619 14:46:58.486999 (+ 1165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:58.487024 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:58.487253 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:58.487466 (+ 213us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:58.488433 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:58.489502 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:58.490433 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:58.490846 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:58.490978 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:58.494311 (+ 3333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:58.494318 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:58.494331 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:58.494355 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:58.496430 (+ 2075us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:58.497056 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:58.497069 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:58.497070 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:58.497145 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:58.497488 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:58.497491 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:58.497493 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:58.497769 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:58.497991 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:58.498189 (+ 198us) server_negotiation.cc:299] Negotiation successful
0619 14:46:58.498333 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":150,"threads_started":1}
I20260619 14:46:58.499136 7033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:58.486109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37899 (local address 127.0.0.1:33476)
0619 14:46:58.486491 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:58.486568 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:58.487014 (+ 446us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:58.487601 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:58.487607 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:58.487967 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:46:58.488247 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:58.488261 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:58.489234 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:58.489243 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:58.490109 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:58.490118 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:58.490685 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:58.490689 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:58.490865 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:58.491561 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:46:58.491588 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:46:58.493760 (+ 2172us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:46:58.495866 (+ 2106us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:58.495878 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:58.495890 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:58.496301 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:58.496628 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:58.496632 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:58.496634 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:58.496825 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:58.497282 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:46:58.497291 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:46:58.497597 (+ 306us) client_negotiation.cc:769] Sending connection context
0619 14:46:58.497792 (+ 195us) client_negotiation.cc:240] Negotiation successful
0619 14:46:58.498142 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":115,"threads_started":1}
I20260619 14:46:58.500969 7029 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36857 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 14:46:58 dist-test-slave-2sb1 krb5kdc[6452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880418, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260619 14:46:58.511852 7033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:58.501989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36857 (local address 127.0.0.1:42554)
0619 14:46:58.502249 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:58.502271 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:58.502376 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:58.502852 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:58.502855 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:58.503043 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:46:58.503162 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:58.503168 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:58.503921 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:58.503924 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:58.504510 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:58.504604 (+ 94us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:58.505834 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:58.505837 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:58.505920 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:58.506521 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:46:58.506537 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:46:58.508096 (+ 1559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:46:58.510036 (+ 1940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:58.510039 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:58.510040 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:58.510281 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:58.510653 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:58.510656 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:58.510657 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:58.510747 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:58.511270 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:46:58.511273 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:46:58.511371 (+ 98us) client_negotiation.cc:769] Sending connection context
0619 14:46:58.511482 (+ 111us) client_negotiation.cc:240] Negotiation successful
0619 14:46:58.511649 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163}
I20260619 14:46:58.511852 7032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:58.502164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42554 (local address 127.0.98.61:36857)
0619 14:46:58.502354 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:58.502359 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:58.502373 (+ 14us) server_negotiation.cc:407] Connection header received
0619 14:46:58.502661 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:58.502664 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:58.502704 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:58.502787 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:58.503287 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:58.503812 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:58.504823 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:58.505700 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:58.505799 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:58.508261 (+ 2462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:58.508264 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:58.508266 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:58.508288 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:58.509895 (+ 1607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:58.510428 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:58.510432 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:58.510433 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:58.510525 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:58.510905 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:58.510908 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:58.510910 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:58.511155 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:58.511247 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:58.511569 (+ 322us) server_negotiation.cc:299] Negotiation successful
0619 14:46:58.511711 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":99}
I20260619 14:46:58.515443 7029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e87b561972740938a4f82c45931705b: Bootstrap starting.
I20260619 14:46:58.518153 7029 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8e87b561972740938a4f82c45931705b: Neither blocks nor log segments found. Creating new log.
I20260619 14:46:58.518923 7029 log.cc:824] T 00000000000000000000000000000000 P 8e87b561972740938a4f82c45931705b: Log is configured to *not* fsync() on all Append() calls
I20260619 14:46:58.521524 7029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e87b561972740938a4f82c45931705b: No bootstrap required, opened a new log
I20260619 14:46:58.524972 7029 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e87b561972740938a4f82c45931705b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a2a5f1050da4c09b1351fb6ecb3986a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37899 } } peers { permanent_uuid: "8e87b561972740938a4f82c45931705b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36857 } }
I20260619 14:46:58.525204 7029 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8e87b561972740938a4f82c45931705b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:46:58.525238 7029 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e87b561972740938a4f82c45931705b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e87b561972740938a4f82c45931705b, State: Initialized, Role: FOLLOWER
I20260619 14:46:58.525718 7029 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e87b561972740938a4f82c45931705b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a2a5f1050da4c09b1351fb6ecb3986a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37899 } } peers { permanent_uuid: "8e87b561972740938a4f82c45931705b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36857 } }
I20260619 14:46:58.526233 7035 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e87b561972740938a4f82c45931705b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a2a5f1050da4c09b1351fb6ecb3986a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37899 } } peers { permanent_uuid: "8e87b561972740938a4f82c45931705b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36857 } } }
I20260619 14:46:58.526362 7035 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e87b561972740938a4f82c45931705b [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:46:58.527065 7029 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e87b561972740938a4f82c45931705b [sys.catalog]: configured and running, proceeding with master startup.
W20260619 14:46:58.532352 7046 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8e87b561972740938a4f82c45931705b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:46:58.532465 7046 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:46:58.534996 6961 master_runner.cc:428] Detected that this master 8e87b561972740938a4f82c45931705b is joining an existing cluster
I20260619 14:46:58.535097 6961 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:36857
I20260619 14:46:58.547495 7032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:58.537861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42556 (local address 127.0.98.61:36857)
0619 14:46:58.537964 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:58.537970 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:58.538301 (+ 331us) server_negotiation.cc:407] Connection header received
0619 14:46:58.538464 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:58.538468 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:58.538511 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:58.538594 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:58.539132 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:58.539813 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:58.540688 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:58.540999 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:58.541099 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:58.542846 (+ 1747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:58.542849 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:58.542852 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:58.542877 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:58.545275 (+ 2398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:58.545814 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:58.545817 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:58.545818 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:58.545893 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:58.546408 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:58.546411 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:58.546413 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:58.546704 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:58.546820 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:58.547109 (+ 289us) server_negotiation.cc:299] Negotiation successful
0619 14:46:58.547326 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":80,"server-negotiator.queue_time_us":41}
I20260619 14:46:58.547519 7052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:58.537570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36857 (local address 127.0.0.1:42556)
0619 14:46:58.538218 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:58.538234 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:58.538357 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:58.538629 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:58.538633 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:58.538880 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:46:58.539010 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:58.539018 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:58.539919 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:58.539922 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:58.540557 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:58.540564 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:58.541132 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:58.541136 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:58.541228 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:58.542055 (+ 827us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:46:58.542074 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:46:58.542741 (+ 667us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:46:58.545425 (+ 2684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:58.545429 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:58.545431 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:58.545696 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:58.546218 (+ 522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:58.546221 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:58.546223 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:58.546300 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:58.546817 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:46:58.546820 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:46:58.546922 (+ 102us) client_negotiation.cc:769] Sending connection context
0619 14:46:58.547048 (+ 126us) client_negotiation.cc:240] Negotiation successful
0619 14:46:58.547233 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"mutex_wait_us":1,"thread_start_us":127,"threads_started":1}
I20260619 14:46:58.548081 6536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:58.537909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33480 (local address 127.0.98.62:37899)
0619 14:46:58.538030 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:58.538034 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:58.538051 (+ 17us) server_negotiation.cc:407] Connection header received
0619 14:46:58.538154 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:58.538157 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:58.538213 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:58.538302 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:58.538881 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:58.539673 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:58.541000 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:58.541300 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:58.541424 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:58.542785 (+ 1361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:58.542789 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:58.542792 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:58.542824 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:58.545293 (+ 2469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:58.546304 (+ 1011us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:58.546308 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:58.546309 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:58.546557 (+ 248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:58.546931 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:58.546935 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:58.546937 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:58.547258 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:58.547361 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:58.547747 (+ 386us) server_negotiation.cc:299] Negotiation successful
0619 14:46:58.547889 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 14:46:58.548579 7053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:58.537405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37899 (local address 127.0.0.1:33480)
0619 14:46:58.537902 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:58.537927 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:58.538036 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:58.538374 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:58.538378 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:58.538639 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:46:58.538767 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:58.538772 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:58.539795 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:58.539799 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:58.540874 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:58.540884 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:58.541414 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:58.541418 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:58.541509 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:58.542091 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:46:58.542108 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:46:58.542629 (+ 521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:46:58.545853 (+ 3224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:58.545856 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:58.545858 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:58.546090 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:58.546694 (+ 604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:58.546697 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:58.546699 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:58.546784 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:58.547410 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:46:58.547413 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:46:58.547500 (+ 87us) client_negotiation.cc:769] Sending connection context
0619 14:46:58.548252 (+ 752us) client_negotiation.cc:240] Negotiation successful
0619 14:46:58.548410 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":404,"mutex_wait_us":68,"thread_start_us":129,"threads_started":1}
Jun 19 14:46:58 dist-test-slave-2sb1 krb5kdc[6452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880417, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260619 14:46:58.564182 7054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:58.553732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36857 (local address 127.0.0.1:42570)
0619 14:46:58.554163 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:58.554205 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:58.554315 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:58.554936 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:58.554942 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:58.555480 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:46:58.555651 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:58.555661 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:58.556813 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:58.556816 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:58.557472 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:58.557480 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:58.557962 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:58.557966 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:58.558087 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:58.558700 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:46:58.558711 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:46:58.560333 (+ 1622us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:46:58.562389 (+ 2056us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:58.562402 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:58.562409 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:58.562686 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:58.562980 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:58.562985 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:58.562986 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:58.563078 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:58.563486 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:46:58.563497 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:46:58.563639 (+ 142us) client_negotiation.cc:769] Sending connection context
0619 14:46:58.563802 (+ 163us) client_negotiation.cc:240] Negotiation successful
0619 14:46:58.564028 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":113,"threads_started":1}
I20260619 14:46:58.564173 7032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:58.554444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42570 (local address 127.0.98.61:36857)
0619 14:46:58.554586 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:58.554589 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:58.554611 (+ 22us) server_negotiation.cc:407] Connection header received
0619 14:46:58.554664 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:58.554666 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:58.554705 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:58.554898 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:58.555782 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:58.556711 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:58.557599 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:58.557834 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:58.557952 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:58.560544 (+ 2592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:58.560554 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:58.560556 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:58.560581 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:58.562240 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:58.562799 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:58.562803 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:58.562805 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:58.562886 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:58.563169 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:58.563174 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:58.563176 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:58.563377 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:58.563484 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:58.563853 (+ 369us) server_negotiation.cc:299] Negotiation successful
0619 14:46:58.563986 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260619 14:46:58.566576 6497 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:36857: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "8e87b561972740938a4f82c45931705b"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 36857
}
attrs {
promote: true
}
}
dest_uuid: "4a2a5f1050da4c09b1351fb6ecb3986a"
cas_config_opid_index: -1
I20260619 14:46:58.567435 6497 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a2a5f1050da4c09b1351fb6ecb3986a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4a2a5f1050da4c09b1351fb6ecb3986a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37899 } } peers { permanent_uuid: "8e87b561972740938a4f82c45931705b" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36857 } attrs { promote: true } }
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
I20260619 14:46:58.569684 7059 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4a2a5f1050da4c09b1351fb6ecb3986a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 8e87b561972740938a4f82c45931705b (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4a2a5f1050da4c09b1351fb6ecb3986a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37899 } } peers { permanent_uuid: "8e87b561972740938a4f82c45931705b" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36857 } attrs { promote: true } } }
I20260619 14:46:58.570719 7003 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e87b561972740938a4f82c45931705b [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:46:58.571841 7059 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:36857
I20260619 14:46:58.572178 7057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a2a5f1050da4c09b1351fb6ecb3986a [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4a2a5f1050da4c09b1351fb6ecb3986a" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4a2a5f1050da4c09b1351fb6ecb3986a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37899 } } peers { permanent_uuid: "8e87b561972740938a4f82c45931705b" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36857 } attrs { promote: true } } }
I20260619 14:46:58.572309 7057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a2a5f1050da4c09b1351fb6ecb3986a [sys.catalog]: This master's current role is: LEADER
I20260619 14:46:58.572513 6961 master.cc:537] Master@127.0.98.61:36857 shutting down...
I20260619 14:46:58.572506 7003 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8e87b561972740938a4f82c45931705b [term 1 FOLLOWER]: Refusing update from remote peer 4a2a5f1050da4c09b1351fb6ecb3986a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260619 14:46:58.573752 7059 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4a2a5f1050da4c09b1351fb6ecb3986a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e87b561972740938a4f82c45931705b" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36857 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 14:46:58.575783 6484 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260619 14:46:58.577777 6961 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 8e87b561972740938a4f82c45931705b [term 1 FOLLOWER]: Raft consensus shutting down.
W20260619 14:46:58.577832 6484 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4a2a5f1050da4c09b1351fb6ecb3986a -> Peer 8e87b561972740938a4f82c45931705b (127.0.98.61:36857): Couldn't send request to peer 8e87b561972740938a4f82c45931705b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260619 14:46:58.578086 6961 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 8e87b561972740938a4f82c45931705b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:46:58.578171 6961 tablet_replica.cc:330] T 00000000000000000000000000000000 P 8e87b561972740938a4f82c45931705b: stopping tablet replica
I20260619 14:46:58.582530 6961 master.cc:552] Master@127.0.98.61:36857 shutdown complete.
I20260619 14:46:58.582615 6961 master_runner.cc:305] Clearing existing system tablet
I20260619 14:46:58.582867 6961 fs_manager.cc:359] Metadata directory not provided
I20260619 14:46:58.582952 6961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:46:58.585013 6961 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260619 14:46:58.586063 7063 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:58.586282 6961 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:46:58.586408 6961 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260619 14:46:58.586476 6961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/master-1/wal
uuid: "8e87b561972740938a4f82c45931705b"
format_stamp: "Formatted at 2026-06-19 14:46:58 on dist-test-slave-2sb1"
server_key: "d865a6b3c25f0d80b8b0e07c39537feb"
server_key_iv: "4f9a5468c1d644355ba10bdbe2ce532d"
server_key_version: "encryptionkey@0"
I20260619 14:46:58.588421 6961 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 8e87b561972740938a4f82c45931705b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:46:58.591692 6961 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 8e87b561972740938a4f82c45931705b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 14:46:58.591786 6961 log.cc:1196] T 00000000000000000000000000000000 P 8e87b561972740938a4f82c45931705b: Deleting WAL directory at /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260619 14:46:58.593228 6961 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 8e87b561972740938a4f82c45931705b: Deleting consensus metadata
I20260619 14:46:58.593534 6961 master_runner.cc:315] Copying system tablet from 127.0.98.62:37899
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260619 14:46:58.594959 6961 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 8e87b561972740938a4f82c45931705b: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:37899
I20260619 14:46:58.595489 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:39275
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37899,127.0.98.61:36857,127.0.98.60:39275
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:46:58.605962 6536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:58.597560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33482 (local address 127.0.98.62:37899)
0619 14:46:58.597695 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:58.597700 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:58.597718 (+ 18us) server_negotiation.cc:407] Connection header received
0619 14:46:58.597868 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:58.597876 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:58.597922 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:58.598017 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:58.598540 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:58.599117 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:58.600466 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:58.600784 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:58.600908 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:58.602187 (+ 1279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:58.602191 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:58.602194 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:58.602222 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:58.603966 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:58.604471 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:58.604483 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:58.604486 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:58.604577 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:58.605001 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:58.605005 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:58.605006 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:58.605225 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:58.605348 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:58.605650 (+ 302us) server_negotiation.cc:299] Negotiation successful
0619 14:46:58.605801 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 14:46:58.605994 7070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:58.597055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37899 (local address 127.0.0.1:33482)
0619 14:46:58.597388 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:58.597406 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:58.597717 (+ 311us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:58.598037 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:58.598040 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:58.598277 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:46:58.598401 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:58.598409 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:58.599249 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:58.599253 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:58.600319 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:58.600330 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:58.600909 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:58.600913 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:58.601036 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:58.601646 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:46:58.601659 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:46:58.602027 (+ 368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:46:58.604101 (+ 2074us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:58.604107 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:58.604110 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:58.604347 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:58.604775 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:58.604781 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:58.604788 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:58.604873 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:58.605337 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:46:58.605340 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:46:58.605458 (+ 118us) client_negotiation.cc:769] Sending connection context
0619 14:46:58.605577 (+ 119us) client_negotiation.cc:240] Negotiation successful
0619 14:46:58.605759 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":137,"threads_started":1}
I20260619 14:46:58.606621 6517 tablet_copy_service.cc:140] P 4a2a5f1050da4c09b1351fb6ecb3986a: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 8e87b561972740938a4f82c45931705b ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:33482)
I20260619 14:46:58.606822 6517 tablet_copy_service.cc:161] P 4a2a5f1050da4c09b1351fb6ecb3986a: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 8e87b561972740938a4f82c45931705b at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:33482: session id = 8e87b561972740938a4f82c45931705b-00000000000000000000000000000000
I20260619 14:46:58.608309 6517 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 4a2a5f1050da4c09b1351fb6ecb3986a: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 14:46:58.610194 6961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 14:46:58.612246 6961 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 8e87b561972740938a4f82c45931705b: tablet copy: Starting download of 0 data blocks...
I20260619 14:46:58.612506 6961 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 8e87b561972740938a4f82c45931705b: tablet copy: Starting download of 1 WAL segments...
I20260619 14:46:58.614720 6961 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 8e87b561972740938a4f82c45931705b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 14:46:58.616791 6517 tablet_copy_service.cc:342] P 4a2a5f1050da4c09b1351fb6ecb3986a: Request end of tablet copy session 8e87b561972740938a4f82c45931705b-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:33482
I20260619 14:46:58.616945 6517 tablet_copy_service.cc:433] P 4a2a5f1050da4c09b1351fb6ecb3986a: ending tablet copy session 8e87b561972740938a4f82c45931705b-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 8e87b561972740938a4f82c45931705b
I20260619 14:46:58.624509 6961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:46:58.626650 7071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:58.626675 7072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:58.626933 7074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:58.628880 6961 server_base.cc:1034] running on GCE node
I20260619 14:46:58.629058 6961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:46:58.629400 6961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:46:58.630622 6961 hybrid_clock.cc:648] HybridClock initialized: now 1781880418630627 us; error 51 us; skew 500 ppm
Jun 19 14:46:58 dist-test-slave-2sb1 krb5kdc[6452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880418, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:46:58.632191 6961 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260619 14:46:58.632740 6961 webserver.cc:460] Webserver started at http://127.0.98.61:39597/ using document root <none> and password file <none>
I20260619 14:46:58.632949 6961 fs_manager.cc:359] Metadata directory not provided
I20260619 14:46:58.633005 6961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:46:58.634186 6961 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260619 14:46:58.635923 7081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:58.636224 6961 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.002s
I20260619 14:46:58.636310 6961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/master-1/wal
uuid: "8e87b561972740938a4f82c45931705b"
format_stamp: "Formatted at 2026-06-19 14:46:58 on dist-test-slave-2sb1"
server_key: "d865a6b3c25f0d80b8b0e07c39537feb"
server_key_iv: "4f9a5468c1d644355ba10bdbe2ce532d"
server_key_version: "encryptionkey@0"
I20260619 14:46:58.636436 6961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:46:58.664959 6961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:46:58.667483 6961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:46:58.673319 6961 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:36857
I20260619 14:46:58.673954 6961 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/master-1/data/info.pb
I20260619 14:46:58.674386 7133 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:36857 every 8 connection(s)
I20260619 14:46:58.675622 7134 sys_catalog.cc:263] Verifying existing consensus state
I20260619 14:46:58.676532 7134 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e87b561972740938a4f82c45931705b: Bootstrap starting.
I20260619 14:46:58.686050 7134 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e87b561972740938a4f82c45931705b: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 14:46:58.686439 7134 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8e87b561972740938a4f82c45931705b: Bootstrap complete.
I20260619 14:46:58.687122 7134 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8e87b561972740938a4f82c45931705b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4a2a5f1050da4c09b1351fb6ecb3986a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37899 } } peers { permanent_uuid: "8e87b561972740938a4f82c45931705b" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36857 } attrs { promote: true } }
I20260619 14:46:58.687278 7134 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8e87b561972740938a4f82c45931705b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8e87b561972740938a4f82c45931705b, State: Initialized, Role: LEARNER
I20260619 14:46:58.687405 7134 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8e87b561972740938a4f82c45931705b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4a2a5f1050da4c09b1351fb6ecb3986a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37899 } } peers { permanent_uuid: "8e87b561972740938a4f82c45931705b" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36857 } attrs { promote: true } }
I20260619 14:46:58.687778 7137 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e87b561972740938a4f82c45931705b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4a2a5f1050da4c09b1351fb6ecb3986a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37899 } } peers { permanent_uuid: "8e87b561972740938a4f82c45931705b" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36857 } attrs { promote: true } } }
I20260619 14:46:58.687966 7137 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e87b561972740938a4f82c45931705b [sys.catalog]: This master's current role is: LEARNER
I20260619 14:46:58.687971 7134 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8e87b561972740938a4f82c45931705b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:46:58.691562 7148 catalog_manager.cc:1259] Loaded cluster ID: 54bc711fead8420ebecba113ea139541
I20260619 14:46:58.691632 7148 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8e87b561972740938a4f82c45931705b: loading cluster ID for follower catalog manager: success
I20260619 14:46:58.693478 7148 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8e87b561972740938a4f82c45931705b: acquiring CA information for follower catalog manager: success
I20260619 14:46:58.694721 7148 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8e87b561972740938a4f82c45931705b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 14:46:58.736876 6688 heartbeater.cc:503] Master 127.0.98.62:37899 was elected leader, sending a full tablet report...
W20260619 14:46:58.760377 7069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:46:58.760826 7069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:46:58.760875 7069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:46:58.765084 7069 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:46:58.765154 7069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:46:58.765174 7069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:46:58.765192 7069 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:46:58.765208 7069 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:46:58.770995 7069 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37899,127.0.98.61:36857,127.0.98.60:39275
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:39275
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7069
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:46:58.772132 7069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:46:58.773227 7069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:46:58.782233 7153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:58.782286 7156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:58.782279 7069 server_base.cc:1034] running on GCE node
W20260619 14:46:58.782487 7154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:58.783099 7069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:46:58.784319 7069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:46:58.785686 7069 hybrid_clock.cc:648] HybridClock initialized: now 1781880418785651 us; error 69 us; skew 500 ppm
Jun 19 14:46:58 dist-test-slave-2sb1 krb5kdc[6452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880418, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:46:58.789522 7069 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260619 14:46:58.790678 7069 webserver.cc:460] Webserver started at http://127.0.98.60:38851/ using document root <none> and password file <none>
I20260619 14:46:58.791307 7069 fs_manager.cc:359] Metadata directory not provided
I20260619 14:46:58.791374 7069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:46:58.791627 7069 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:46:58.793993 7069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/master-2/data/instance:
uuid: "768cb9d7dfb349e3b12896639c2ddd4f"
format_stamp: "Formatted at 2026-06-19 14:46:58 on dist-test-slave-2sb1"
server_key: "27ff1effcac04ef53d768d5f4a0379f1"
server_key_iv: "9e8a94eb2ad24ea3b3f112cad8b7821a"
server_key_version: "encryptionkey@0"
I20260619 14:46:58.794617 7069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/master-2/wal/instance:
uuid: "768cb9d7dfb349e3b12896639c2ddd4f"
format_stamp: "Formatted at 2026-06-19 14:46:58 on dist-test-slave-2sb1"
server_key: "27ff1effcac04ef53d768d5f4a0379f1"
server_key_iv: "9e8a94eb2ad24ea3b3f112cad8b7821a"
server_key_version: "encryptionkey@0"
I20260619 14:46:58.799005 7069 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 14:46:58.801805 7163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:58.803736 7069 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:46:58.803898 7069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/master-2/wal
uuid: "768cb9d7dfb349e3b12896639c2ddd4f"
format_stamp: "Formatted at 2026-06-19 14:46:58 on dist-test-slave-2sb1"
server_key: "27ff1effcac04ef53d768d5f4a0379f1"
server_key_iv: "9e8a94eb2ad24ea3b3f112cad8b7821a"
server_key_version: "encryptionkey@0"
I20260619 14:46:58.804020 7069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:46:58.825677 7069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:46:58.829659 7069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:46:58.829914 7069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:46:58.838719 7069 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:39275
I20260619 14:46:58.839074 7215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:39275 every 8 connection(s)
I20260619 14:46:58.839825 7069 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/master-2/data/info.pb
I20260619 14:46:58.842900 7216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:46:58.844264 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 7069
I20260619 14:46:58.847455 7216 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37899 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 14:46:58 dist-test-slave-2sb1 krb5kdc[6452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880417, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Jun 19 14:46:58 dist-test-slave-2sb1 krb5kdc[6452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880418, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 14:46:58.861406 6536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:58.849751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53776 (local address 127.0.98.62:37899)
0619 14:46:58.849872 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:58.849876 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:58.850564 (+ 688us) server_negotiation.cc:407] Connection header received
0619 14:46:58.850717 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:58.850720 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:58.850775 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:58.850850 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:58.851867 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:58.852562 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:58.853606 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:58.853907 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:58.854024 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:58.857312 (+ 3288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:58.857316 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:58.857319 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:58.857351 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:58.859069 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:58.859694 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:58.859697 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:58.859698 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:58.859805 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:58.860183 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:58.860186 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:58.860187 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:58.860388 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:58.860494 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:58.861078 (+ 584us) server_negotiation.cc:299] Negotiation successful
0619 14:46:58.861211 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 14:46:58.862290 7219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:58.849554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37899 (local address 127.0.0.1:53776)
0619 14:46:58.850406 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:58.850459 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:58.850589 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:58.850936 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:58.850942 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:58.851503 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:46:58.851692 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:58.851701 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:58.852757 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:58.852760 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:58.853478 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:58.853486 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:58.854109 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:58.854114 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:58.854253 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:58.854950 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:46:58.854976 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:46:58.857130 (+ 2154us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:46:58.859218 (+ 2088us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:58.859228 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:58.859240 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:58.859578 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:58.859972 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:58.859975 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:58.859977 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:58.860071 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:58.860609 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:46:58.860615 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:46:58.860828 (+ 213us) client_negotiation.cc:769] Sending connection context
0619 14:46:58.861022 (+ 194us) client_negotiation.cc:240] Negotiation successful
0619 14:46:58.862049 (+ 1027us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":221,"mutex_wait_us":722,"thread_start_us":120,"threads_started":1}
I20260619 14:46:58.862300 7218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:58.846041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34528 (local address 127.0.98.60:39275)
0619 14:46:58.846534 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:58.846554 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:58.846599 (+ 45us) server_negotiation.cc:407] Connection header received
0619 14:46:58.847469 (+ 870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:58.847497 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:58.848048 (+ 551us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:58.848434 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:58.849729 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:58.851703 (+ 1974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:58.852637 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:58.853275 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:58.853632 (+ 357us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:58.856717 (+ 3085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:58.856723 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:58.856735 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:58.856757 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:58.858776 (+ 2019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:58.859347 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:58.859353 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:58.859354 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:58.859464 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:58.859786 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:58.859789 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:58.859791 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:58.860390 (+ 599us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:58.860575 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:58.861153 (+ 578us) server_negotiation.cc:299] Negotiation successful
0619 14:46:58.861380 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":56,"server-negotiator.queue_time_us":340,"thread_start_us":143,"threads_started":1}
I20260619 14:46:58.863837 7216 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36857 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 14:46:58 dist-test-slave-2sb1 krb5kdc[6452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880418, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260619 14:46:58.874306 7219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:58.864425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36857 (local address 127.0.0.1:57908)
0619 14:46:58.864616 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:58.864633 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:58.864747 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:58.865112 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:58.865115 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:58.865272 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:46:58.865375 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:58.865380 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:58.866406 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:58.866409 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:58.867015 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:58.867023 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:58.867728 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:58.867732 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:58.867810 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:58.868555 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:46:58.868574 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:46:58.870102 (+ 1528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:46:58.872404 (+ 2302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:58.872408 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:58.872410 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:58.872825 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:58.873194 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:58.873197 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:58.873198 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:58.873296 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:58.873872 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:46:58.873875 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:46:58.873964 (+ 89us) client_negotiation.cc:769] Sending connection context
0619 14:46:58.874048 (+ 84us) client_negotiation.cc:240] Negotiation successful
0619 14:46:58.874172 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116}
I20260619 14:46:58.874429 7220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:58.864544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57908 (local address 127.0.98.61:36857)
0619 14:46:58.864851 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:58.864855 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:58.864867 (+ 12us) server_negotiation.cc:407] Connection header received
0619 14:46:58.864903 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:58.864905 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:58.864941 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:58.865018 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:58.865606 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:58.866262 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:58.867150 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:58.867530 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:58.867638 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:58.870272 (+ 2634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:58.870276 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:58.870279 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:58.870307 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:58.872233 (+ 1926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:58.872957 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:58.872960 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:58.872961 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:58.873069 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:58.873450 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:58.873453 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:58.873455 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:58.873763 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:58.873846 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:58.874131 (+ 285us) server_negotiation.cc:299] Negotiation successful
0619 14:46:58.874251 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":120,"threads_started":1}
I20260619 14:46:58.875465 7216 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39275 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 14:46:58 dist-test-slave-2sb1 krb5kdc[6452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880418, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260619 14:46:58.885218 7219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:58.875956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39275 (local address 127.0.0.1:34542)
0619 14:46:58.876084 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:58.876108 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:58.876175 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:58.876413 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:58.876415 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:58.876552 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:46:58.876625 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:58.876630 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:58.877863 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:58.877866 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:58.878599 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:58.878608 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:58.879240 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:58.879243 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:58.879307 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:58.879938 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:46:58.879954 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:46:58.881623 (+ 1669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:46:58.883428 (+ 1805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:58.883431 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:58.883433 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:58.883673 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:58.884072 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:58.884075 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:58.884076 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:58.884172 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:58.884568 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:46:58.884570 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:46:58.884859 (+ 289us) client_negotiation.cc:769] Sending connection context
0619 14:46:58.884953 (+ 94us) client_negotiation.cc:240] Negotiation successful
0619 14:46:58.885074 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260619 14:46:58.885401 7218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:58.876102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34542 (local address 127.0.98.60:39275)
0619 14:46:58.876224 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:58.876228 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:58.876239 (+ 11us) server_negotiation.cc:407] Connection header received
0619 14:46:58.876269 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:58.876271 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:58.876305 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:58.876397 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:58.876776 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:58.877733 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:58.878799 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:58.879087 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:58.879186 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:58.881790 (+ 2604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:58.881793 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:58.881795 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:58.881817 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:58.883307 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:58.883820 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:58.883823 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:58.883824 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:58.883918 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:58.884277 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:58.884280 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:58.884281 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:58.884475 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:58.884562 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:58.885019 (+ 457us) server_negotiation.cc:299] Negotiation successful
0619 14:46:58.885190 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 14:46:58.888756 7216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 768cb9d7dfb349e3b12896639c2ddd4f: Bootstrap starting.
I20260619 14:46:58.891274 7216 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 768cb9d7dfb349e3b12896639c2ddd4f: Neither blocks nor log segments found. Creating new log.
I20260619 14:46:58.892036 7216 log.cc:824] T 00000000000000000000000000000000 P 768cb9d7dfb349e3b12896639c2ddd4f: Log is configured to *not* fsync() on all Append() calls
I20260619 14:46:58.894253 7216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 768cb9d7dfb349e3b12896639c2ddd4f: No bootstrap required, opened a new log
I20260619 14:46:58.897581 7216 raft_consensus.cc:348] T 00000000000000000000000000000000 P 768cb9d7dfb349e3b12896639c2ddd4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a2a5f1050da4c09b1351fb6ecb3986a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37899 } } peers { permanent_uuid: "8e87b561972740938a4f82c45931705b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36857 } } peers { permanent_uuid: "768cb9d7dfb349e3b12896639c2ddd4f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39275 } }
I20260619 14:46:58.897832 7216 raft_consensus.cc:374] T 00000000000000000000000000000000 P 768cb9d7dfb349e3b12896639c2ddd4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:46:58.897917 7216 raft_consensus.cc:729] T 00000000000000000000000000000000 P 768cb9d7dfb349e3b12896639c2ddd4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 768cb9d7dfb349e3b12896639c2ddd4f, State: Initialized, Role: FOLLOWER
I20260619 14:46:58.898423 7216 consensus_queue.cc:260] T 00000000000000000000000000000000 P 768cb9d7dfb349e3b12896639c2ddd4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a2a5f1050da4c09b1351fb6ecb3986a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37899 } } peers { permanent_uuid: "8e87b561972740938a4f82c45931705b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36857 } } peers { permanent_uuid: "768cb9d7dfb349e3b12896639c2ddd4f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39275 } }
I20260619 14:46:58.899011 7222 sys_catalog.cc:455] T 00000000000000000000000000000000 P 768cb9d7dfb349e3b12896639c2ddd4f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4a2a5f1050da4c09b1351fb6ecb3986a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37899 } } peers { permanent_uuid: "8e87b561972740938a4f82c45931705b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36857 } } peers { permanent_uuid: "768cb9d7dfb349e3b12896639c2ddd4f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39275 } } }
I20260619 14:46:58.899129 7222 sys_catalog.cc:458] T 00000000000000000000000000000000 P 768cb9d7dfb349e3b12896639c2ddd4f [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:46:58.899652 7216 sys_catalog.cc:564] T 00000000000000000000000000000000 P 768cb9d7dfb349e3b12896639c2ddd4f [sys.catalog]: configured and running, proceeding with master startup.
W20260619 14:46:58.905027 7233 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 768cb9d7dfb349e3b12896639c2ddd4f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:46:58.905143 7233 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:46:58.909228 7069 master_runner.cc:428] Detected that this master 768cb9d7dfb349e3b12896639c2ddd4f is joining an existing cluster
I20260619 14:46:58.909319 7069 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:39275
I20260619 14:46:58.915840 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 6559
I20260619 14:46:58.923271 7240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:58.911587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39275 (local address 127.0.0.1:34546)
0619 14:46:58.912999 (+ 1412us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:58.913012 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:58.913128 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:58.913407 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:58.913410 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:58.913691 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:46:58.913819 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:58.913827 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:58.914748 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:58.914754 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:58.915596 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:58.915606 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:58.917433 (+ 1827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:58.917439 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:58.917573 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:58.918309 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:46:58.918330 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:46:58.918856 (+ 526us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:46:58.921193 (+ 2337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:58.921197 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:58.921199 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:58.921471 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:58.921836 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:58.921840 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:58.921842 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:58.921967 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:58.922535 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:46:58.922539 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:46:58.922644 (+ 105us) client_negotiation.cc:769] Sending connection context
0619 14:46:58.922767 (+ 123us) client_negotiation.cc:240] Negotiation successful
0619 14:46:58.923053 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1353,"mutex_wait_us":29,"thread_start_us":508,"threads_started":1}
I20260619 14:46:58.923319 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:37475
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:37899
--webserver_port=35971
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:37899,127.0.98.61:36857
--tserver_master_addrs=127.0.98.62:37899,127.0.98.61:36857,127.0.98.60:39275 with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:46:58.923928 7218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:58.912358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34546 (local address 127.0.98.60:39275)
0619 14:46:58.912462 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:58.912466 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:58.913074 (+ 608us) server_negotiation.cc:407] Connection header received
0619 14:46:58.913231 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:58.913239 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:58.913285 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:58.913382 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:58.913938 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:58.914665 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:58.915707 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:58.916033 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:58.916125 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:58.919012 (+ 2887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:58.919019 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:58.919025 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:58.919054 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:58.921060 (+ 2006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:58.921604 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:58.921610 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:58.921615 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:58.921733 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:58.922080 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:58.922086 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:58.922091 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:58.922372 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:58.922482 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:58.922844 (+ 362us) server_negotiation.cc:299] Negotiation successful
0619 14:46:58.923154 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":122,"server-negotiator.queue_time_us":44}
I20260619 14:46:58.931313 7220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:58.912825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57916 (local address 127.0.98.61:36857)
0619 14:46:58.912952 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:58.912957 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:58.923550 (+ 10593us) server_negotiation.cc:407] Connection header received
0619 14:46:58.923721 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:58.923724 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:58.923779 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:58.923829 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:58.924397 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:58.925229 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:58.926417 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:58.926655 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:58.926742 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:58.927992 (+ 1250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:58.927995 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:58.927997 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:58.928024 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:58.929568 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:58.930115 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:58.930119 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:58.930120 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:58.930197 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:58.930506 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:58.930509 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:58.930510 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:58.930702 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:58.930780 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:58.931052 (+ 272us) server_negotiation.cc:299] Negotiation successful
0619 14:46:58.931199 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 14:46:58.931324 7240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:58.912704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36857 (local address 127.0.0.1:57916)
0619 14:46:58.923466 (+ 10762us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:58.923482 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:58.923620 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:58.923973 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:58.923976 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:58.924160 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:46:58.924266 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:58.924272 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:58.925370 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:58.925373 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:58.926293 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:58.926302 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:58.926781 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:58.926784 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:58.926889 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:58.927371 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:46:58.927385 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:46:58.927850 (+ 465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:46:58.929693 (+ 1843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:58.929696 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:58.929699 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:58.929963 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:58.930327 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:58.930330 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:58.930332 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:58.930421 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:58.930792 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:46:58.930795 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:46:58.930894 (+ 99us) client_negotiation.cc:769] Sending connection context
0619 14:46:58.931009 (+ 115us) client_negotiation.cc:240] Negotiation successful
0619 14:46:58.931148 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10697,"thread_start_us":99,"threads_started":1}
I20260619 14:46:58.942440 7239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:58.911501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37899 (local address 127.0.0.1:53786)
0619 14:46:58.912280 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:58.912298 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:58.912438 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:58.912995 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:58.912999 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:58.913297 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:46:58.913445 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:58.913452 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:58.914513 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:58.914516 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:58.915646 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:58.915659 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:58.917600 (+ 1941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:58.917603 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:58.917701 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:58.937435 (+ 19734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:46:58.937477 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:46:58.937970 (+ 493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:46:58.940515 (+ 2545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:58.940528 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:58.940531 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:58.940882 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:58.941255 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:58.941258 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:58.941260 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:58.941352 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:58.941825 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:46:58.941828 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:46:58.941935 (+ 107us) client_negotiation.cc:769] Sending connection context
0619 14:46:58.942047 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 14:46:58.942212 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":696,"mutex_wait_us":3,"thread_start_us":117,"threads_started":1}
I20260619 14:46:58.943025 6536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:58.911594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53786 (local address 127.0.98.62:37899)
0619 14:46:58.911727 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:58.911731 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:58.912394 (+ 663us) server_negotiation.cc:407] Connection header received
0619 14:46:58.912702 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:58.912707 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:58.912770 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:58.912829 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:58.913549 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:58.914393 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:58.915813 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:58.916119 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:58.916188 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:58.938214 (+ 22026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:58.938222 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:58.938229 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:58.938277 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:58.940333 (+ 2056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:58.941033 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:58.941037 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:58.941039 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:58.941131 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:58.941457 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:58.941461 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:58.941463 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:58.941689 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:58.941782 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:58.942738 (+ 956us) server_negotiation.cc:299] Negotiation successful
0619 14:46:58.942866 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
Jun 19 14:46:58 dist-test-slave-2sb1 krb5kdc[6452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880417, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260619 14:46:58.957461 7054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:58.947168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39275 (local address 127.0.0.1:34558)
0619 14:46:58.947332 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:58.947346 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:58.947417 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:58.947704 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:58.947707 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:58.947907 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:46:58.948025 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:58.948032 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:58.949106 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:58.949112 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:58.950021 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:58.950033 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:58.950696 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:58.950700 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:58.950802 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:58.951360 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:46:58.951375 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:46:58.953128 (+ 1753us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:46:58.955346 (+ 2218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:58.955350 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:58.955352 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:58.955731 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:58.956111 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:58.956114 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:58.956116 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:58.956213 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:58.956866 (+ 653us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:46:58.956871 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:46:58.956985 (+ 114us) client_negotiation.cc:769] Sending connection context
0619 14:46:58.957103 (+ 118us) client_negotiation.cc:240] Negotiation successful
0619 14:46:58.957274 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94}
I20260619 14:46:58.957466 7218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:58.947284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34558 (local address 127.0.98.60:39275)
0619 14:46:58.947394 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:58.947399 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:58.947413 (+ 14us) server_negotiation.cc:407] Connection header received
0619 14:46:58.947529 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:58.947532 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:58.947583 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:58.947671 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:58.948156 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:58.948970 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:58.950176 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:58.950550 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:58.950678 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:58.953321 (+ 2643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:58.953325 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:58.953328 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:58.953361 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:58.955197 (+ 1836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:58.955898 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:58.955902 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:58.955904 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:58.955999 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:58.956325 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:58.956330 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:58.956335 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:58.956695 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:58.956859 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:58.957177 (+ 318us) server_negotiation.cc:299] Negotiation successful
0619 14:46:58.957308 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 14:46:58.959077 6497 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:39275: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "768cb9d7dfb349e3b12896639c2ddd4f"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 39275
}
attrs {
promote: true
}
}
dest_uuid: "4a2a5f1050da4c09b1351fb6ecb3986a"
cas_config_opid_index: 5
I20260619 14:46:58.959769 6497 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a2a5f1050da4c09b1351fb6ecb3986a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4a2a5f1050da4c09b1351fb6ecb3986a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37899 } } peers { permanent_uuid: "8e87b561972740938a4f82c45931705b" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36857 } attrs { promote: true } } peers { permanent_uuid: "768cb9d7dfb349e3b12896639c2ddd4f" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39275 } attrs { promote: true } }
I20260619 14:46:58.960942 7057 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4a2a5f1050da4c09b1351fb6ecb3986a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 768cb9d7dfb349e3b12896639c2ddd4f (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4a2a5f1050da4c09b1351fb6ecb3986a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37899 } } peers { permanent_uuid: "8e87b561972740938a4f82c45931705b" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36857 } attrs { promote: true } } peers { permanent_uuid: "768cb9d7dfb349e3b12896639c2ddd4f" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39275 } attrs { promote: true } } }
I20260619 14:46:58.962517 7057 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:39275
I20260619 14:46:58.962706 7057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a2a5f1050da4c09b1351fb6ecb3986a [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4a2a5f1050da4c09b1351fb6ecb3986a" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4a2a5f1050da4c09b1351fb6ecb3986a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37899 } } peers { permanent_uuid: "8e87b561972740938a4f82c45931705b" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36857 } attrs { promote: true } } peers { permanent_uuid: "768cb9d7dfb349e3b12896639c2ddd4f" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39275 } attrs { promote: true } } }
I20260619 14:46:58.962796 7057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a2a5f1050da4c09b1351fb6ecb3986a [sys.catalog]: This master's current role is: LEADER
I20260619 14:46:58.963757 7190 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 768cb9d7dfb349e3b12896639c2ddd4f [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:46:58.965284 7190 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 768cb9d7dfb349e3b12896639c2ddd4f [term 1 FOLLOWER]: Refusing update from remote peer 4a2a5f1050da4c09b1351fb6ecb3986a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260619 14:46:58.966228 7059 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4a2a5f1050da4c09b1351fb6ecb3986a [LEADER]: Connected to new peer: Peer: permanent_uuid: "768cb9d7dfb349e3b12896639c2ddd4f" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39275 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:46:58.966619 7220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:58.961295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57930 (local address 127.0.98.61:36857)
0619 14:46:58.961580 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:58.961586 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:58.961854 (+ 268us) server_negotiation.cc:407] Connection header received
0619 14:46:58.961907 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:58.961912 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:58.961982 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:58.962068 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:46:58.962425 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:58.963272 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:58.964894 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:58.965662 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:58.965769 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:58.965931 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:58.966361 (+ 430us) server_negotiation.cc:299] Negotiation successful
0619 14:46:58.966486 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199}
I20260619 14:46:58.966980 7054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:58.961065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36857 (local address 127.0.0.1:57930)
0619 14:46:58.961676 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:58.961691 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:58.961773 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:58.962199 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:58.962202 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:58.962218 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:46:58.962310 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:58.962315 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:58.963420 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:58.963423 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:58.964740 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:58.964753 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:58.966010 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:58.966015 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:58.966116 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:58.966169 (+ 53us) client_negotiation.cc:769] Sending connection context
0619 14:46:58.966244 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 14:46:58.966856 (+ 612us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538}
I20260619 14:46:58.967559 7108 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8e87b561972740938a4f82c45931705b [term 1 LEARNER]: Refusing update from remote peer 4a2a5f1050da4c09b1351fb6ecb3986a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260619 14:46:58.967619 7069 master.cc:537] Master@127.0.98.60:39275 shutting down...
I20260619 14:46:58.968113 7059 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4a2a5f1050da4c09b1351fb6ecb3986a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e87b561972740938a4f82c45931705b" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36857 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20260619 14:46:58.969409 6482 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4a2a5f1050da4c09b1351fb6ecb3986a -> Peer 768cb9d7dfb349e3b12896639c2ddd4f (127.0.98.60:39275): Couldn't send request to peer 768cb9d7dfb349e3b12896639c2ddd4f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260619 14:46:58.970058 7108 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8e87b561972740938a4f82c45931705b [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 768cb9d7dfb349e3b12896639c2ddd4f (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4a2a5f1050da4c09b1351fb6ecb3986a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37899 } } peers { permanent_uuid: "8e87b561972740938a4f82c45931705b" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36857 } attrs { promote: true } } peers { permanent_uuid: "768cb9d7dfb349e3b12896639c2ddd4f" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39275 } attrs { promote: true } } }
I20260619 14:46:58.970408 6822 heartbeater.cc:503] Master 127.0.98.62:37899 was elected leader, sending a full tablet report...
I20260619 14:46:58.972013 7137 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e87b561972740938a4f82c45931705b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4a2a5f1050da4c09b1351fb6ecb3986a" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4a2a5f1050da4c09b1351fb6ecb3986a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37899 } } peers { permanent_uuid: "8e87b561972740938a4f82c45931705b" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36857 } attrs { promote: true } } peers { permanent_uuid: "768cb9d7dfb349e3b12896639c2ddd4f" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39275 } attrs { promote: true } } }
I20260619 14:46:58.972144 7137 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e87b561972740938a4f82c45931705b [sys.catalog]: This master's current role is: LEARNER
I20260619 14:46:58.972343 7244 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e87b561972740938a4f82c45931705b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a2a5f1050da4c09b1351fb6ecb3986a. Latest consensus state: current_term: 1 leader_uuid: "4a2a5f1050da4c09b1351fb6ecb3986a" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4a2a5f1050da4c09b1351fb6ecb3986a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37899 } } peers { permanent_uuid: "8e87b561972740938a4f82c45931705b" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36857 } attrs { promote: true } } peers { permanent_uuid: "768cb9d7dfb349e3b12896639c2ddd4f" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39275 } attrs { promote: true } } }
I20260619 14:46:58.972474 7244 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e87b561972740938a4f82c45931705b [sys.catalog]: This master's current role is: LEARNER
I20260619 14:46:58.974795 7069 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 768cb9d7dfb349e3b12896639c2ddd4f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:46:58.974893 7059 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 4a2a5f1050da4c09b1351fb6ecb3986a: attempting to promote NON_VOTER 8e87b561972740938a4f82c45931705b to VOTER
I20260619 14:46:58.975097 7069 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 768cb9d7dfb349e3b12896639c2ddd4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:46:58.975194 7069 tablet_replica.cc:330] T 00000000000000000000000000000000 P 768cb9d7dfb349e3b12896639c2ddd4f: stopping tablet replica
I20260619 14:46:58.975364 7059 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a2a5f1050da4c09b1351fb6ecb3986a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4a2a5f1050da4c09b1351fb6ecb3986a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37899 } } peers { permanent_uuid: "8e87b561972740938a4f82c45931705b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36857 } attrs { promote: false } } peers { permanent_uuid: "768cb9d7dfb349e3b12896639c2ddd4f" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39275 } attrs { promote: true } }
I20260619 14:46:58.977875 7108 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8e87b561972740938a4f82c45931705b [term 1 LEARNER]: Refusing update from remote peer 4a2a5f1050da4c09b1351fb6ecb3986a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260619 14:46:58.978826 7057 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4a2a5f1050da4c09b1351fb6ecb3986a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e87b561972740938a4f82c45931705b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36857 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260619 14:46:58.979055 6482 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4a2a5f1050da4c09b1351fb6ecb3986a -> Peer 768cb9d7dfb349e3b12896639c2ddd4f (127.0.98.60:39275): Couldn't send request to peer 768cb9d7dfb349e3b12896639c2ddd4f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260619 14:46:58.980671 7057 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4a2a5f1050da4c09b1351fb6ecb3986a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 8e87b561972740938a4f82c45931705b (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4a2a5f1050da4c09b1351fb6ecb3986a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37899 } } peers { permanent_uuid: "8e87b561972740938a4f82c45931705b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36857 } attrs { promote: false } } peers { permanent_uuid: "768cb9d7dfb349e3b12896639c2ddd4f" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39275 } attrs { promote: true } } }
I20260619 14:46:58.982205 7057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a2a5f1050da4c09b1351fb6ecb3986a [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4a2a5f1050da4c09b1351fb6ecb3986a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4a2a5f1050da4c09b1351fb6ecb3986a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37899 } } peers { permanent_uuid: "8e87b561972740938a4f82c45931705b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36857 } attrs { promote: false } } peers { permanent_uuid: "768cb9d7dfb349e3b12896639c2ddd4f" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39275 } attrs { promote: true } } }
I20260619 14:46:58.982326 7057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a2a5f1050da4c09b1351fb6ecb3986a [sys.catalog]: This master's current role is: LEADER
I20260619 14:46:58.982455 7057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a2a5f1050da4c09b1351fb6ecb3986a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4a2a5f1050da4c09b1351fb6ecb3986a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4a2a5f1050da4c09b1351fb6ecb3986a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37899 } } peers { permanent_uuid: "8e87b561972740938a4f82c45931705b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36857 } attrs { promote: false } } peers { permanent_uuid: "768cb9d7dfb349e3b12896639c2ddd4f" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39275 } attrs { promote: true } } }
I20260619 14:46:58.982534 7057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a2a5f1050da4c09b1351fb6ecb3986a [sys.catalog]: This master's current role is: LEADER
I20260619 14:46:58.982697 7069 master.cc:552] Master@127.0.98.60:39275 shutdown complete.
I20260619 14:46:58.982791 7069 master_runner.cc:305] Clearing existing system tablet
I20260619 14:46:58.983107 7069 fs_manager.cc:359] Metadata directory not provided
I20260619 14:46:58.983212 7069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:46:58.983392 7108 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8e87b561972740938a4f82c45931705b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 8e87b561972740938a4f82c45931705b (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4a2a5f1050da4c09b1351fb6ecb3986a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37899 } } peers { permanent_uuid: "8e87b561972740938a4f82c45931705b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36857 } attrs { promote: false } } peers { permanent_uuid: "768cb9d7dfb349e3b12896639c2ddd4f" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39275 } attrs { promote: true } } }
I20260619 14:46:58.985065 7069 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 14:46:58.985363 7244 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e87b561972740938a4f82c45931705b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4a2a5f1050da4c09b1351fb6ecb3986a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4a2a5f1050da4c09b1351fb6ecb3986a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37899 } } peers { permanent_uuid: "8e87b561972740938a4f82c45931705b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36857 } attrs { promote: false } } peers { permanent_uuid: "768cb9d7dfb349e3b12896639c2ddd4f" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39275 } attrs { promote: true } } }
I20260619 14:46:58.985478 7244 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e87b561972740938a4f82c45931705b [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:46:58.986244 7247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:58.986528 7069 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:46:58.986675 7069 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 14:46:58.986716 7069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/master-2/wal
uuid: "768cb9d7dfb349e3b12896639c2ddd4f"
format_stamp: "Formatted at 2026-06-19 14:46:58 on dist-test-slave-2sb1"
server_key: "27ff1effcac04ef53d768d5f4a0379f1"
server_key_iv: "9e8a94eb2ad24ea3b3f112cad8b7821a"
server_key_version: "encryptionkey@0"
I20260619 14:46:58.989776 7069 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 768cb9d7dfb349e3b12896639c2ddd4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:46:58.992868 7069 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 768cb9d7dfb349e3b12896639c2ddd4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260619 14:46:58.993089 7069 log.cc:1196] T 00000000000000000000000000000000 P 768cb9d7dfb349e3b12896639c2ddd4f: Deleting WAL directory at /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260619 14:46:58.993495 7069 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 768cb9d7dfb349e3b12896639c2ddd4f: Deleting consensus metadata
I20260619 14:46:58.993731 7069 master_runner.cc:315] Copying system tablet from 127.0.98.62:37899
I20260619 14:46:58.995086 7069 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 768cb9d7dfb349e3b12896639c2ddd4f: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:37899
I20260619 14:46:59.005916 6536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:58.995923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53788 (local address 127.0.98.62:37899)
0619 14:46:58.996071 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:58.996078 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:58.996220 (+ 142us) server_negotiation.cc:407] Connection header received
0619 14:46:58.996377 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:58.996380 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:58.996450 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:58.996566 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:58.997281 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:58.998187 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:58.999469 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:58.999874 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:59.000009 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:59.001608 (+ 1599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:59.001612 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:59.001615 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:59.001650 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:59.003945 (+ 2295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:59.004525 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:59.004529 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:59.004531 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:59.004631 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:59.004966 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:59.004970 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:59.004972 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:59.005213 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:59.005338 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:59.005618 (+ 280us) server_negotiation.cc:299] Negotiation successful
0619 14:46:59.005750 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 14:46:59.005920 7253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:58.995785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37899 (local address 127.0.0.1:53788)
0619 14:46:58.996109 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:58.996125 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:58.996243 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:58.996599 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:58.996602 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:58.996852 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:46:58.996972 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:58.996981 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:58.998358 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:58.998362 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:58.999314 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:58.999326 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:59.000006 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:59.000010 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:59.000094 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:59.000853 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:46:59.000869 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:46:59.001369 (+ 500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:46:59.004103 (+ 2734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:59.004107 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:59.004109 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:59.004405 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:59.004752 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:59.004755 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:59.004758 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:59.004861 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:59.005324 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:46:59.005327 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:46:59.005432 (+ 105us) client_negotiation.cc:769] Sending connection context
0619 14:46:59.005541 (+ 109us) client_negotiation.cc:240] Negotiation successful
0619 14:46:59.005702 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":92,"threads_started":1}
I20260619 14:46:59.006899 6517 tablet_copy_service.cc:140] P 4a2a5f1050da4c09b1351fb6ecb3986a: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 768cb9d7dfb349e3b12896639c2ddd4f ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:53788)
I20260619 14:46:59.007014 6517 tablet_copy_service.cc:161] P 4a2a5f1050da4c09b1351fb6ecb3986a: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 768cb9d7dfb349e3b12896639c2ddd4f at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:53788: session id = 768cb9d7dfb349e3b12896639c2ddd4f-00000000000000000000000000000000
I20260619 14:46:59.008186 6517 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 4a2a5f1050da4c09b1351fb6ecb3986a: Tablet Copy: opened 0 blocks and 1 log segments
I20260619 14:46:59.009862 7069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260619 14:46:59.013001 7069 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 768cb9d7dfb349e3b12896639c2ddd4f: tablet copy: Starting download of 0 data blocks...
I20260619 14:46:59.013393 7069 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 768cb9d7dfb349e3b12896639c2ddd4f: tablet copy: Starting download of 1 WAL segments...
I20260619 14:46:59.015491 7069 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 768cb9d7dfb349e3b12896639c2ddd4f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260619 14:46:59.018462 6517 tablet_copy_service.cc:342] P 4a2a5f1050da4c09b1351fb6ecb3986a: Request end of tablet copy session 768cb9d7dfb349e3b12896639c2ddd4f-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:53788
I20260619 14:46:59.018576 6517 tablet_copy_service.cc:433] P 4a2a5f1050da4c09b1351fb6ecb3986a: ending tablet copy session 768cb9d7dfb349e3b12896639c2ddd4f-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 768cb9d7dfb349e3b12896639c2ddd4f
I20260619 14:46:59.027608 7069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:46:59.029747 7254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:59.030011 7257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:59.032866 7255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:59.033542 7069 server_base.cc:1034] running on GCE node
I20260619 14:46:59.033752 7069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:46:59.034103 7069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:46:59.035274 7069 hybrid_clock.cc:648] HybridClock initialized: now 1781880419035264 us; error 42 us; skew 500 ppm
Jun 19 14:46:59 dist-test-slave-2sb1 krb5kdc[6452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880419, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:46:59.037292 7069 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260619 14:46:59.038048 7069 webserver.cc:460] Webserver started at http://127.0.98.60:43965/ using document root <none> and password file <none>
I20260619 14:46:59.038265 7069 fs_manager.cc:359] Metadata directory not provided
I20260619 14:46:59.038311 7069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:46:59.039780 7069 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260619 14:46:59.040937 7264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:59.041270 7069 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260619 14:46:59.041353 7069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/master-2/wal
uuid: "768cb9d7dfb349e3b12896639c2ddd4f"
format_stamp: "Formatted at 2026-06-19 14:46:58 on dist-test-slave-2sb1"
server_key: "27ff1effcac04ef53d768d5f4a0379f1"
server_key_iv: "9e8a94eb2ad24ea3b3f112cad8b7821a"
server_key_version: "encryptionkey@0"
I20260619 14:46:59.041443 7069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:46:59.080722 7069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:46:59.083206 7069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:46:59.089355 7069 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:39275
I20260619 14:46:59.090188 7069 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/master-2/data/info.pb
I20260619 14:46:59.090562 7316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:39275 every 8 connection(s)
I20260619 14:46:59.092440 7317 sys_catalog.cc:263] Verifying existing consensus state
I20260619 14:46:59.093847 7317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 768cb9d7dfb349e3b12896639c2ddd4f: Bootstrap starting.
I20260619 14:46:59.107728 7317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 768cb9d7dfb349e3b12896639c2ddd4f: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 14:46:59.108139 7317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 768cb9d7dfb349e3b12896639c2ddd4f: Bootstrap complete.
I20260619 14:46:59.108839 7317 raft_consensus.cc:348] T 00000000000000000000000000000000 P 768cb9d7dfb349e3b12896639c2ddd4f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4a2a5f1050da4c09b1351fb6ecb3986a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37899 } } peers { permanent_uuid: "8e87b561972740938a4f82c45931705b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36857 } attrs { promote: false } } peers { permanent_uuid: "768cb9d7dfb349e3b12896639c2ddd4f" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39275 } attrs { promote: true } }
I20260619 14:46:59.108995 7317 raft_consensus.cc:729] T 00000000000000000000000000000000 P 768cb9d7dfb349e3b12896639c2ddd4f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 768cb9d7dfb349e3b12896639c2ddd4f, State: Initialized, Role: LEARNER
I20260619 14:46:59.109100 7317 consensus_queue.cc:260] T 00000000000000000000000000000000 P 768cb9d7dfb349e3b12896639c2ddd4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4a2a5f1050da4c09b1351fb6ecb3986a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37899 } } peers { permanent_uuid: "8e87b561972740938a4f82c45931705b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36857 } attrs { promote: false } } peers { permanent_uuid: "768cb9d7dfb349e3b12896639c2ddd4f" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39275 } attrs { promote: true } }
I20260619 14:46:59.109620 7317 sys_catalog.cc:564] T 00000000000000000000000000000000 P 768cb9d7dfb349e3b12896639c2ddd4f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:46:59.109607 7323 sys_catalog.cc:455] T 00000000000000000000000000000000 P 768cb9d7dfb349e3b12896639c2ddd4f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4a2a5f1050da4c09b1351fb6ecb3986a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37899 } } peers { permanent_uuid: "8e87b561972740938a4f82c45931705b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36857 } attrs { promote: false } } peers { permanent_uuid: "768cb9d7dfb349e3b12896639c2ddd4f" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39275 } attrs { promote: true } } }
I20260619 14:46:59.109752 7323 sys_catalog.cc:458] T 00000000000000000000000000000000 P 768cb9d7dfb349e3b12896639c2ddd4f [sys.catalog]: This master's current role is: LEARNER
W20260619 14:46:59.110661 7242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:46:59.111095 7242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:46:59.111212 7242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260619 14:46:59.113112 7334 catalog_manager.cc:1259] Loaded cluster ID: 54bc711fead8420ebecba113ea139541
I20260619 14:46:59.113178 7334 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 768cb9d7dfb349e3b12896639c2ddd4f: loading cluster ID for follower catalog manager: success
I20260619 14:46:59.115023 7334 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 768cb9d7dfb349e3b12896639c2ddd4f: acquiring CA information for follower catalog manager: success
I20260619 14:46:59.116201 7334 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 768cb9d7dfb349e3b12896639c2ddd4f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260619 14:46:59.117105 7242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:46:59.117168 7242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:46:59.117286 7242 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 14:46:59.124279 7242 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:37475
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=35971
--tserver_master_addrs=127.0.98.62:37899,127.0.98.61:36857,127.0.98.60:39275
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7242
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:46:59.125537 7242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:46:59.126751 7242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:46:59.139074 7336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:59.139025 7339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:59.139034 7337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:59.139807 7242 server_base.cc:1034] running on GCE node
I20260619 14:46:59.140345 7242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:46:59.141166 7242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:46:59.142391 7242 hybrid_clock.cc:648] HybridClock initialized: now 1781880419142370 us; error 44 us; skew 500 ppm
Jun 19 14:46:59 dist-test-slave-2sb1 krb5kdc[6452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880419, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:46:59.146010 7242 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260619 14:46:59.147042 7242 webserver.cc:460] Webserver started at http://127.0.98.1:35971/ using document root <none> and password file <none>
I20260619 14:46:59.147783 7242 fs_manager.cc:359] Metadata directory not provided
I20260619 14:46:59.147912 7242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:46:59.152323 7242 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 14:46:59.154865 7346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:59.156602 7242 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 14:46:59.156806 7242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/ts-0/wal
uuid: "1b116e7a13764f34b66cdbb5e53338e4"
format_stamp: "Formatted at 2026-06-19 14:46:57 on dist-test-slave-2sb1"
server_key: "896c6fadbbc1d7ac99ea7218c02cc486"
server_key_iv: "1aab366100b84b6a7657b0cd27c053d2"
server_key_version: "encryptionkey@0"
I20260619 14:46:59.157439 7242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:46:59.190732 7242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:46:59.194821 7242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:46:59.195075 7242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:46:59.195835 7242 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:46:59.196835 7242 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:46:59.196906 7242 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:59.196976 7242 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:46:59.197023 7242 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:59.204214 6956 heartbeater.cc:503] Master 127.0.98.62:37899 was elected leader, sending a full tablet report...
I20260619 14:46:59.207556 7242 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37475
I20260619 14:46:59.207594 7459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37475 every 8 connection(s)
I20260619 14:46:59.208603 7242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/ts-0/data/info.pb
I20260619 14:46:59.215890 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 7242
I20260619 14:46:59.216087 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 6693
Jun 19 14:46:59 dist-test-slave-2sb1 krb5kdc[6452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880419, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Jun 19 14:46:59 dist-test-slave-2sb1 krb5kdc[6452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880419, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 14:46:59.223495 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:37385
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:37899
--webserver_port=44951
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:37899,127.0.98.61:36857
--tserver_master_addrs=127.0.98.62:37899,127.0.98.61:36857,127.0.98.60:39275 with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 14:46:59 dist-test-slave-2sb1 krb5kdc[6452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880419, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260619 14:46:59.228973 6536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:59.211171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33597 (local address 127.0.98.62:37899)
0619 14:46:59.211462 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:59.211467 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:59.212634 (+ 1167us) server_negotiation.cc:407] Connection header received
0619 14:46:59.213592 (+ 958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:59.213595 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:59.213654 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:59.213768 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:59.215290 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:59.216068 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:59.218268 (+ 2200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:59.218616 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:59.218696 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:59.223373 (+ 4677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:59.223380 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:59.223386 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:59.223424 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:59.226572 (+ 3148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:59.227123 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:59.227130 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:59.227135 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:59.227230 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:59.227563 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:59.227570 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:59.227574 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:59.227823 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:59.227913 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:59.228540 (+ 627us) server_negotiation.cc:299] Negotiation successful
0619 14:46:59.228729 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193}
I20260619 14:46:59.228961 7220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:59.210927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47323 (local address 127.0.98.61:36857)
0619 14:46:59.211340 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:59.211345 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:59.212580 (+ 1235us) server_negotiation.cc:407] Connection header received
0619 14:46:59.213638 (+ 1058us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:59.213644 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:59.213715 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:59.213767 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:59.215298 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:59.215855 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:59.218257 (+ 2402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:59.218616 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:59.218696 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:59.223395 (+ 4699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:59.223402 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:59.223408 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:59.223442 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:59.225708 (+ 2266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:59.226535 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:59.226542 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:59.226548 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:59.226657 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:59.227209 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:59.227215 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:59.227220 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:59.227531 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:59.227635 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:59.228547 (+ 912us) server_negotiation.cc:299] Negotiation successful
0619 14:46:59.228738 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312}
I20260619 14:46:59.230988 7465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:59.211396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36857 (local address 127.0.98.1:47323)
0619 14:46:59.212469 (+ 1073us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:59.212479 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:59.213473 (+ 994us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:59.213997 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:59.214000 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:59.214536 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:46:59.215142 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:59.215151 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:59.216042 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:59.216047 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:59.218090 (+ 2043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:59.218102 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:59.218918 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:59.218922 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:59.219257 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:59.220184 (+ 927us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:46:59.220217 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:46:59.222732 (+ 2515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:46:59.225863 (+ 3131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:59.225874 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:59.225899 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:59.226369 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:59.226799 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:59.226802 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:59.226804 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:59.227064 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:59.227721 (+ 657us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:46:59.227730 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:46:59.228194 (+ 464us) client_negotiation.cc:769] Sending connection context
0619 14:46:59.229171 (+ 977us) client_negotiation.cc:240] Negotiation successful
0619 14:46:59.229640 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":977,"mutex_wait_us":252,"spinlock_wait_cycles":100992,"thread_start_us":181,"threads_started":1}
I20260619 14:46:59.231156 7467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:59.211648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44093 (local address 127.0.98.60:39275)
0619 14:46:59.211910 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:59.211914 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:59.212329 (+ 415us) server_negotiation.cc:407] Connection header received
0619 14:46:59.213727 (+ 1398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:59.213730 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:59.213788 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:59.213845 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:59.220860 (+ 7015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:59.222201 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:59.223419 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:59.223809 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:59.223903 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:59.226884 (+ 2981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:59.226889 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:59.226892 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:59.226923 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:59.229065 (+ 2142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:59.229640 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:59.229644 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:59.229646 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:59.229741 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:59.230078 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:59.230081 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:59.230083 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:59.230322 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:59.230403 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:59.230817 (+ 414us) server_negotiation.cc:299] Negotiation successful
0619 14:46:59.230959 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":82,"threads_started":1}
I20260619 14:46:59.232059 7464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:59.211396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39275 (local address 127.0.98.1:44093)
0619 14:46:59.212168 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:59.212222 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:59.213605 (+ 1383us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:59.213986 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:59.213989 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:59.214531 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:46:59.215144 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:59.215151 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:59.222391 (+ 7240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:59.222398 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:59.223236 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:59.223282 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:59.224118 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:59.224124 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:59.224251 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:59.224847 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:46:59.224871 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:46:59.226701 (+ 1830us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:46:59.229209 (+ 2508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:59.229212 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:59.229214 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:59.229490 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:59.229862 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:59.229865 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:59.229867 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:59.229951 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:59.230485 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:46:59.230488 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:46:59.230614 (+ 126us) client_negotiation.cc:769] Sending connection context
0619 14:46:59.231684 (+ 1070us) client_negotiation.cc:240] Negotiation successful
0619 14:46:59.231852 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"mutex_wait_us":69,"spinlock_wait_cycles":12160,"thread_start_us":182,"threads_started":1}
I20260619 14:46:59.232407 7466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:59.211396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37899 (local address 127.0.98.1:33597)
0619 14:46:59.212552 (+ 1156us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:59.212564 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:59.213334 (+ 770us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:59.213960 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:59.213967 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:59.214502 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:46:59.215129 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:59.215151 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:59.217279 (+ 2128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:59.217285 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:59.218100 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:59.218105 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:59.218918 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:59.218922 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:59.219259 (+ 337us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:59.220262 (+ 1003us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:46:59.220274 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:46:59.223184 (+ 2910us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:46:59.226703 (+ 3519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:59.226706 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:59.226708 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:59.226998 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:59.227343 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:59.227349 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:59.227354 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:59.227445 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:59.227976 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:46:59.227981 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:46:59.228185 (+ 204us) client_negotiation.cc:769] Sending connection context
0619 14:46:59.229923 (+ 1738us) client_negotiation.cc:240] Negotiation successful
0619 14:46:59.230113 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1057,"mutex_wait_us":351,"spinlock_wait_cycles":12416,"thread_start_us":101,"threads_started":1}
I20260619 14:46:59.232898 7462 heartbeater.cc:347] Connected to a master server at 127.0.98.61:36857
I20260619 14:46:59.232977 7461 heartbeater.cc:347] Connected to a master server at 127.0.98.60:39275
I20260619 14:46:59.233183 7461 heartbeater.cc:464] Registering TS with master...
I20260619 14:46:59.233193 7462 heartbeater.cc:464] Registering TS with master...
I20260619 14:46:59.233810 7460 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37899
I20260619 14:46:59.233856 7461 heartbeater.cc:511] Master 127.0.98.60:39275 requested a full tablet report, sending...
I20260619 14:46:59.233860 7462 heartbeater.cc:511] Master 127.0.98.61:36857 requested a full tablet report, sending...
I20260619 14:46:59.233904 7460 heartbeater.cc:464] Registering TS with master...
I20260619 14:46:59.234628 7460 heartbeater.cc:511] Master 127.0.98.62:37899 requested a full tablet report, sending...
I20260619 14:46:59.235532 7281 ts_manager.cc:194] Registered new tserver with Master: 1b116e7a13764f34b66cdbb5e53338e4 (127.0.98.1:37475)
I20260619 14:46:59.236022 7098 ts_manager.cc:194] Registered new tserver with Master: 1b116e7a13764f34b66cdbb5e53338e4 (127.0.98.1:37475)
I20260619 14:46:59.236362 6497 ts_manager.cc:194] Re-registered known tserver with Master: 1b116e7a13764f34b66cdbb5e53338e4 (127.0.98.1:37475)
I20260619 14:46:59.237118 6497 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33597
W20260619 14:46:59.357585 7468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:46:59.357920 7468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:46:59.357999 7468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:46:59.362689 7468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:46:59.362766 7468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:46:59.362856 7468 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 14:46:59.368819 7468 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:37385
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=44951
--tserver_master_addrs=127.0.98.62:37899,127.0.98.61:36857,127.0.98.60:39275
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7468
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:46:59.369998 7468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:46:59.371058 7468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:46:59.381734 7473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:59.382040 7468 server_base.cc:1034] running on GCE node
W20260619 14:46:59.382120 7474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:59.381664 7476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:59.382779 7468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:46:59.385185 7468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:46:59.388820 7468 hybrid_clock.cc:648] HybridClock initialized: now 1781880419388690 us; error 160 us; skew 500 ppm
Jun 19 14:46:59 dist-test-slave-2sb1 krb5kdc[6452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880419, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:46:59.392629 7468 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260619 14:46:59.393818 7468 webserver.cc:460] Webserver started at http://127.0.98.2:44951/ using document root <none> and password file <none>
I20260619 14:46:59.394455 7468 fs_manager.cc:359] Metadata directory not provided
I20260619 14:46:59.394575 7468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:46:59.399081 7468 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 14:46:59.401721 7483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:59.403514 7468 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 14:46:59.403682 7468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/ts-1/wal
uuid: "0f513b290a8c49ee81a89fa8d2152c04"
format_stamp: "Formatted at 2026-06-19 14:46:57 on dist-test-slave-2sb1"
server_key: "337f40618b3e9f51a1c1e87bd723c09c"
server_key_iv: "5cc90915ec433651079b3418b4af4b63"
server_key_version: "encryptionkey@0"
I20260619 14:46:59.404577 7468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:46:59.405265 7054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:59.400985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39275 (local address 127.0.0.1:34568)
0619 14:46:59.401097 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:59.401125 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:59.401227 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:59.401579 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:59.401583 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:59.401609 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:46:59.401751 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:59.401762 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:59.402813 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:59.402819 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:59.403724 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:59.403736 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:59.404764 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:59.404768 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:59.404884 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:59.404927 (+ 43us) client_negotiation.cc:769] Sending connection context
0619 14:46:59.404996 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 14:46:59.405150 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260619 14:46:59.405905 7467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:59.401074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34568 (local address 127.0.98.60:39275)
0619 14:46:59.401207 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:59.401212 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:59.401229 (+ 17us) server_negotiation.cc:407] Connection header received
0619 14:46:59.401347 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:59.401352 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:59.401437 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:59.401558 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:46:59.401925 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:59.402683 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:59.403907 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:59.404528 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:59.405583 (+ 1055us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:59.405708 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:59.405743 (+ 35us) server_negotiation.cc:299] Negotiation successful
0619 14:46:59.405809 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 14:46:59.406554 7291 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 768cb9d7dfb349e3b12896639c2ddd4f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260619 14:46:59.406952 7323 sys_catalog.cc:455] T 00000000000000000000000000000000 P 768cb9d7dfb349e3b12896639c2ddd4f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a2a5f1050da4c09b1351fb6ecb3986a. Latest consensus state: current_term: 1 leader_uuid: "4a2a5f1050da4c09b1351fb6ecb3986a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4a2a5f1050da4c09b1351fb6ecb3986a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37899 } } peers { permanent_uuid: "8e87b561972740938a4f82c45931705b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36857 } attrs { promote: false } } peers { permanent_uuid: "768cb9d7dfb349e3b12896639c2ddd4f" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39275 } attrs { promote: true } } }
I20260619 14:46:59.407137 7323 sys_catalog.cc:458] T 00000000000000000000000000000000 P 768cb9d7dfb349e3b12896639c2ddd4f [sys.catalog]: This master's current role is: LEARNER
I20260619 14:46:59.433212 7468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:46:59.437929 7468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:46:59.438169 7468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:46:59.439102 7468 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:46:59.440632 7468 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:46:59.440738 7468 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:59.440809 7468 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:46:59.440847 7468 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:59.455163 7468 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37385
I20260619 14:46:59.455276 7596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37385 every 8 connection(s)
I20260619 14:46:59.456899 7468 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/ts-1/data/info.pb
I20260619 14:46:59.463165 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 7468
I20260619 14:46:59.463382 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 6827
I20260619 14:46:59.471787 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:45697
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:37899
--webserver_port=35361
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:37899,127.0.98.61:36857
--tserver_master_addrs=127.0.98.62:37899,127.0.98.61:36857,127.0.98.60:39275 with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 14:46:59 dist-test-slave-2sb1 krb5kdc[6452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880419, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Jun 19 14:46:59 dist-test-slave-2sb1 krb5kdc[6452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880419, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Jun 19 14:46:59 dist-test-slave-2sb1 krb5kdc[6452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880419, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260619 14:46:59.482231 7220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:59.460134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58597 (local address 127.0.98.61:36857)
0619 14:46:59.460303 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:59.460310 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:59.461594 (+ 1284us) server_negotiation.cc:407] Connection header received
0619 14:46:59.462951 (+ 1357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:59.462955 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:59.463019 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:59.463082 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:59.466313 (+ 3231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:59.467216 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:59.469127 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:59.469466 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:59.469546 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:59.477098 (+ 7552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:59.477106 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:59.477113 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:59.477154 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:59.479375 (+ 2221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:59.480140 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:59.480144 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:59.480146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:59.480243 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:59.480746 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:59.480750 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:59.480752 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:59.481045 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:59.481137 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:59.481885 (+ 748us) server_negotiation.cc:299] Negotiation successful
0619 14:46:59.482051 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 14:46:59.482280 6536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:59.461347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47197 (local address 127.0.98.62:37899)
0619 14:46:59.461466 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:59.461470 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:59.464955 (+ 3485us) server_negotiation.cc:407] Connection header received
0619 14:46:59.465145 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:59.465149 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:59.465204 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:59.465259 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:59.465946 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:59.466966 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:59.470853 (+ 3887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:59.471269 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:59.471367 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:59.476741 (+ 5374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:59.476748 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:59.476755 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:59.476800 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:59.480460 (+ 3660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:59.480989 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:59.480993 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:59.480994 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:59.481084 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:59.481379 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:59.481382 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:59.481383 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:59.481591 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:59.481675 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:59.482034 (+ 359us) server_negotiation.cc:299] Negotiation successful
0619 14:46:59.482148 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 14:46:59.482398 7467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:59.459406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49627 (local address 127.0.98.60:39275)
0619 14:46:59.459549 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:59.459555 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:59.461589 (+ 2034us) server_negotiation.cc:407] Connection header received
0619 14:46:59.462954 (+ 1365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:59.462958 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:59.463019 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:59.463082 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:59.467117 (+ 4035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:59.467867 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:59.469118 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:59.469466 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:59.469546 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:59.476230 (+ 6684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:59.476235 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:59.476238 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:59.476273 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:59.479148 (+ 2875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:59.480191 (+ 1043us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:59.480199 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:59.480202 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:59.480317 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:59.480785 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:59.480792 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:59.480797 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:59.481118 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:59.481231 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:59.481957 (+ 726us) server_negotiation.cc:299] Negotiation successful
0619 14:46:59.482140 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 14:46:59.484200 7601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:59.460459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37899 (local address 127.0.98.2:47197)
0619 14:46:59.464841 (+ 4382us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:59.464860 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:59.465026 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:59.465417 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:59.465420 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:59.465640 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:46:59.465761 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:59.465767 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:59.469878 (+ 4111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:59.469881 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:59.470690 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:59.470699 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:59.471552 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:59.471557 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:59.471677 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:59.472318 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:46:59.472332 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:46:59.476496 (+ 4164us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:46:59.480598 (+ 4102us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:59.480601 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:59.480604 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:59.480873 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:59.481194 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:59.481197 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:59.481199 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:59.481279 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:59.481728 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:46:59.481731 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:46:59.481837 (+ 106us) client_negotiation.cc:769] Sending connection context
0619 14:46:59.482568 (+ 731us) client_negotiation.cc:240] Negotiation successful
0619 14:46:59.483022 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4294,"mutex_wait_us":27,"thread_start_us":106,"threads_started":1}
I20260619 14:46:59.484200 7603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:59.459966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36857 (local address 127.0.98.2:58597)
0619 14:46:59.461396 (+ 1430us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:59.461464 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:59.462648 (+ 1184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:59.464145 (+ 1497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:59.464157 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:59.464890 (+ 733us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:46:59.465695 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:59.465717 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:59.467406 (+ 1689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:59.467409 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:59.468962 (+ 1553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:59.468973 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:59.469772 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:59.469776 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:59.470048 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:59.475020 (+ 4972us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:46:59.475042 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:46:59.476874 (+ 1832us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:46:59.479497 (+ 2623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:59.479503 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:59.479508 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:59.480001 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:59.480369 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:59.480374 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:59.480376 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:59.480584 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:59.481205 (+ 621us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:46:59.481212 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:46:59.481588 (+ 376us) client_negotiation.cc:769] Sending connection context
0619 14:46:59.482501 (+ 913us) client_negotiation.cc:240] Negotiation successful
0619 14:46:59.483076 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1181,"mutex_wait_us":229,"spinlock_wait_cycles":19456,"thread_start_us":174,"threads_started":1}
I20260619 14:46:59.485684 7599 heartbeater.cc:347] Connected to a master server at 127.0.98.61:36857
I20260619 14:46:59.485823 7602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:59.459964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39275 (local address 127.0.98.2:49627)
0619 14:46:59.461454 (+ 1490us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:59.461468 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:59.462648 (+ 1180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:59.464559 (+ 1911us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:59.464564 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:59.465964 (+ 1400us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:46:59.466089 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:59.466096 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:59.468010 (+ 1914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:59.468013 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:59.468965 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:59.468973 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:59.469772 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:59.469776 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:59.470684 (+ 908us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:59.471445 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:46:59.471472 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:46:59.475956 (+ 4484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:46:59.479367 (+ 3411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:59.479379 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:59.479407 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:59.480009 (+ 602us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:59.480468 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:59.480473 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:59.480478 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:59.480586 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:59.481306 (+ 720us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:46:59.481312 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:46:59.481590 (+ 278us) client_negotiation.cc:769] Sending connection context
0619 14:46:59.483288 (+ 1698us) client_negotiation.cc:240] Negotiation successful
0619 14:46:59.483542 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1400,"mutex_wait_us":1097,"spinlock_wait_cycles":31616,"thread_start_us":112,"threads_started":1}
I20260619 14:46:59.485947 7599 heartbeater.cc:464] Registering TS with master...
I20260619 14:46:59.486590 7599 heartbeater.cc:511] Master 127.0.98.61:36857 requested a full tablet report, sending...
I20260619 14:46:59.486910 7598 heartbeater.cc:347] Connected to a master server at 127.0.98.60:39275
I20260619 14:46:59.487008 7598 heartbeater.cc:464] Registering TS with master...
I20260619 14:46:59.487174 7597 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37899
I20260619 14:46:59.487190 7598 heartbeater.cc:511] Master 127.0.98.60:39275 requested a full tablet report, sending...
I20260619 14:46:59.487248 7597 heartbeater.cc:464] Registering TS with master...
I20260619 14:46:59.487385 7597 heartbeater.cc:511] Master 127.0.98.62:37899 requested a full tablet report, sending...
I20260619 14:46:59.487987 7098 ts_manager.cc:194] Registered new tserver with Master: 0f513b290a8c49ee81a89fa8d2152c04 (127.0.98.2:37385)
I20260619 14:46:59.488139 6497 ts_manager.cc:194] Re-registered known tserver with Master: 0f513b290a8c49ee81a89fa8d2152c04 (127.0.98.2:37385)
I20260619 14:46:59.488770 7281 ts_manager.cc:194] Registered new tserver with Master: 0f513b290a8c49ee81a89fa8d2152c04 (127.0.98.2:37385)
I20260619 14:46:59.488989 6497 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47197
W20260619 14:46:59.616452 7604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:46:59.616874 7604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:46:59.616966 7604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:46:59.622048 7604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:46:59.622123 7604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:46:59.622229 7604 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 14:46:59.628841 7604 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:45697
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=35361
--tserver_master_addrs=127.0.98.62:37899,127.0.98.61:36857,127.0.98.60:39275
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7604
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:46:59.629989 7604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:46:59.631055 7604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:46:59.641631 7612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:59.641759 7604 server_base.cc:1034] running on GCE node
W20260619 14:46:59.641637 7609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:46:59.641666 7610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:46:59.642464 7604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:46:59.643227 7604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:46:59.644446 7604 hybrid_clock.cc:648] HybridClock initialized: now 1781880419644426 us; error 37 us; skew 500 ppm
Jun 19 14:46:59 dist-test-slave-2sb1 krb5kdc[6452](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880419, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:46:59.648170 7604 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260619 14:46:59.649394 7604 webserver.cc:460] Webserver started at http://127.0.98.3:35361/ using document root <none> and password file <none>
I20260619 14:46:59.650029 7604 fs_manager.cc:359] Metadata directory not provided
I20260619 14:46:59.650151 7604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:46:59.654781 7604 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:46:59.657472 7619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:59.659075 7604 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:46:59.659266 7604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/ts-2/wal
uuid: "2cb8830304e145db8591b8ff765eeef3"
format_stamp: "Formatted at 2026-06-19 14:46:58 on dist-test-slave-2sb1"
server_key: "158f7d962baf155821fe6fd4f3400e36"
server_key_iv: "8ef0555896dbc59b5aaba4d9047079f7"
server_key_version: "encryptionkey@0"
I20260619 14:46:59.659883 7604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:46:59.674805 7604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:46:59.678462 7604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:46:59.678711 7604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:46:59.679324 7604 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:46:59.680269 7604 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:46:59.680343 7604 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:59.680416 7604 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:46:59.680470 7604 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:46:59.690647 7604 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45697
I20260619 14:46:59.690692 7732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45697 every 8 connection(s)
I20260619 14:46:59.691843 7604 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/ts-2/data/info.pb
I20260619 14:46:59.692629 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 7604
Jun 19 14:46:59 dist-test-slave-2sb1 krb5kdc[6452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880419, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Jun 19 14:46:59 dist-test-slave-2sb1 krb5kdc[6452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880419, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Jun 19 14:46:59 dist-test-slave-2sb1 krb5kdc[6452](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880419, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260619 14:46:59.711585 7746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:59.696439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57936 (local address 127.0.98.61:36857)
0619 14:46:59.698447 (+ 2008us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:59.698451 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:59.698464 (+ 13us) server_negotiation.cc:407] Connection header received
0619 14:46:59.698505 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:59.698509 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:59.698558 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:59.699458 (+ 900us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:59.700429 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:59.701273 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:59.702899 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:59.703264 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:59.703365 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:59.706669 (+ 3304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:59.706674 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:59.706679 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:59.706712 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:59.708546 (+ 1834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:59.709392 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:59.709398 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:59.709403 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:59.709494 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:59.709809 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:59.709813 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:59.709814 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:59.710145 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:59.710280 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:59.711009 (+ 729us) server_negotiation.cc:299] Negotiation successful
0619 14:46:59.711178 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":843,"server-negotiator.queue_time_us":1944,"thread_start_us":95,"threads_started":1}
I20260619 14:46:59.711797 6536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:59.694926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40951 (local address 127.0.98.62:37899)
0619 14:46:59.695358 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:59.695364 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:59.695777 (+ 413us) server_negotiation.cc:407] Connection header received
0619 14:46:59.697939 (+ 2162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:59.697942 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:59.697989 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:59.698037 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:59.699469 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:59.700232 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:59.702112 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:59.702445 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:59.702517 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:59.706472 (+ 3955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:59.706478 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:59.706484 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:59.706512 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:59.708913 (+ 2401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:59.709708 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:59.709714 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:59.709717 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:59.709813 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:59.710309 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:59.710314 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:59.710317 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:59.710577 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:59.710685 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:59.711446 (+ 761us) server_negotiation.cc:299] Negotiation successful
0619 14:46:59.711576 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339}
I20260619 14:46:59.713233 7467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:59.694780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35211 (local address 127.0.98.60:39275)
0619 14:46:59.695241 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:59.695246 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:59.696746 (+ 1500us) server_negotiation.cc:407] Connection header received
0619 14:46:59.697276 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:59.697283 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:59.697345 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:59.697401 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:59.702616 (+ 5215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:59.703380 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:59.705105 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:59.705414 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:59.705502 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:59.709169 (+ 3667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:59.709176 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:59.709182 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:59.709215 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:59.710837 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:59.711309 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:59.711313 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:59.711314 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:59.711397 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:59.712229 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:59.712232 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:59.712233 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:59.712443 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:59.712523 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:59.712924 (+ 401us) server_negotiation.cc:299] Negotiation successful
0619 14:46:59.713058 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368}
I20260619 14:46:59.713595 7744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:59.694449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37899 (local address 127.0.98.3:40951)
0619 14:46:59.695627 (+ 1178us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:59.695683 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:59.697408 (+ 1725us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:59.698158 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:59.698161 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:59.698411 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:46:59.699302 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:59.699323 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:59.700391 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:59.700394 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:59.701982 (+ 1588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:59.701992 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:59.702728 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:59.702732 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:59.702974 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:59.703787 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:46:59.703814 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:46:59.706261 (+ 2447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:46:59.709071 (+ 2810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:59.709079 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:59.709116 (+ 37us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:59.709558 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:59.709942 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:59.709946 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:59.709949 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:59.710168 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:59.710757 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:46:59.710765 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:46:59.711134 (+ 369us) client_negotiation.cc:769] Sending connection context
0619 14:46:59.711988 (+ 854us) client_negotiation.cc:240] Negotiation successful
0619 14:46:59.712378 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":940,"mutex_wait_us":91,"thread_start_us":749,"threads_started":1}
I20260619 14:46:59.714211 7742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:59.694450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39275 (local address 127.0.98.3:35211)
0619 14:46:59.696632 (+ 2182us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:59.696672 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:59.697027 (+ 355us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:59.697718 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:59.697721 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:59.698410 (+ 689us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:46:59.702501 (+ 4091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:59.702507 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:59.703529 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:59.703534 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:59.704475 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:59.704488 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:59.705692 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:59.705698 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:59.705806 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:59.706901 (+ 1095us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:46:59.706921 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:46:59.708968 (+ 2047us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:46:59.710953 (+ 1985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:59.710956 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:59.710958 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:59.711195 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:59.711504 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:59.711507 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:59.711508 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:59.711602 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:59.712600 (+ 998us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:46:59.712605 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:46:59.712727 (+ 122us) client_negotiation.cc:769] Sending connection context
0619 14:46:59.713619 (+ 892us) client_negotiation.cc:240] Negotiation successful
0619 14:46:59.714015 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2077,"mutex_wait_us":4393,"spinlock_wait_cycles":6400,"thread_start_us":101,"threads_started":1}
I20260619 14:46:59.714483 7220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:59.695963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:32841 (local address 127.0.98.61:36857)
0619 14:46:59.696115 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:59.696130 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:59.696596 (+ 466us) server_negotiation.cc:407] Connection header received
0619 14:46:59.697333 (+ 737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:59.697336 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:59.697386 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:59.697477 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:59.699478 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:59.700245 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:59.704617 (+ 4372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:59.704994 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:59.705088 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:59.708749 (+ 3661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:59.708755 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:59.708759 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:59.708788 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:59.711068 (+ 2280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:59.713078 (+ 2010us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:59.713081 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:59.713083 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:59.713209 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:59.713521 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:59.713524 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:59.713526 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:59.713747 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:59.713850 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:59.714194 (+ 344us) server_negotiation.cc:299] Negotiation successful
0619 14:46:59.714340 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":77}
I20260619 14:46:59.715013 7734 heartbeater.cc:347] Connected to a master server at 127.0.98.60:39275
I20260619 14:46:59.715082 7733 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37899
I20260619 14:46:59.715294 7733 heartbeater.cc:464] Registering TS with master...
I20260619 14:46:59.715757 7743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:59.695014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53802 (local address 127.0.98.62:37899)
0619 14:46:59.695901 (+ 887us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:59.695905 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:59.695915 (+ 10us) server_negotiation.cc:407] Connection header received
0619 14:46:59.695958 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:59.695961 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:59.696022 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:59.697288 (+ 1266us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:59.697402 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:59.698180 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:59.699712 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:59.700064 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:59.700578 (+ 514us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:59.703525 (+ 2947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:59.703530 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:59.703536 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:59.703565 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:59.706457 (+ 2892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:59.712943 (+ 6486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:59.712946 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:59.712947 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:59.713035 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:59.713387 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:59.713390 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:59.713392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:59.713669 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:59.715385 (+ 1716us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:59.715464 (+ 79us) server_negotiation.cc:299] Negotiation successful
0619 14:46:59.715613 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":824,"thread_start_us":86,"threads_started":1}
I20260619 14:46:59.715894 7733 heartbeater.cc:511] Master 127.0.98.62:37899 requested a full tablet report, sending...
I20260619 14:46:59.716588 7748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:59.697063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34574 (local address 127.0.98.60:39275)
0619 14:46:59.698306 (+ 1243us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:59.698313 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:59.699710 (+ 1397us) server_negotiation.cc:407] Connection header received
0619 14:46:59.699873 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:59.699876 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:59.699923 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:59.699979 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:59.701347 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:59.702185 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:59.704088 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:59.704450 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:59.704544 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:59.708038 (+ 3494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:46:59.708046 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:46:59.708050 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:46:59.708083 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:46:59.712610 (+ 4527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:59.715197 (+ 2587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:59.715201 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:59.715204 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:59.715302 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:46:59.715621 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:46:59.715624 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:46:59.715626 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:46:59.715842 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:46:59.715971 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:59.716275 (+ 304us) server_negotiation.cc:299] Negotiation successful
0619 14:46:59.716419 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1161,"thread_start_us":81,"threads_started":1}
I20260619 14:46:59.715296 7734 heartbeater.cc:464] Registering TS with master...
I20260619 14:46:59.717114 7734 heartbeater.cc:511] Master 127.0.98.60:39275 requested a full tablet report, sending...
I20260619 14:46:59.715022 7745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:59.694804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36857 (local address 127.0.98.3:32841)
0619 14:46:59.696312 (+ 1508us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:59.696327 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:59.696928 (+ 601us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:59.697710 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:59.697719 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:59.698596 (+ 877us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:46:59.699301 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:59.699323 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:59.701430 (+ 2107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:59.701433 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:59.702346 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:59.702354 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:59.705256 (+ 2902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:59.705259 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:59.705351 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:59.706256 (+ 905us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:46:59.706276 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:46:59.708584 (+ 2308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:46:59.712737 (+ 4153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:59.712740 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:59.712742 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:59.712966 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:59.713336 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:46:59.713338 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:46:59.713340 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:46:59.713423 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:46:59.713906 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:46:59.713909 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:46:59.714005 (+ 96us) client_negotiation.cc:769] Sending connection context
0619 14:46:59.714673 (+ 668us) client_negotiation.cc:240] Negotiation successful
0619 14:46:59.714821 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1442,"mutex_wait_us":119,"spinlock_wait_cycles":39168,"thread_start_us":165,"threads_started":1}
I20260619 14:46:59.717417 6497 ts_manager.cc:194] Re-registered known tserver with Master: 2cb8830304e145db8591b8ff765eeef3 (127.0.98.3:45697)
I20260619 14:46:59.718127 6497 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40951
I20260619 14:46:59.718255 7735 heartbeater.cc:347] Connected to a master server at 127.0.98.61:36857
I20260619 14:46:59.718336 7735 heartbeater.cc:464] Registering TS with master...
I20260619 14:46:59.718510 7735 heartbeater.cc:511] Master 127.0.98.61:36857 requested a full tablet report, sending...
I20260619 14:46:59.719039 7281 ts_manager.cc:194] Registered new tserver with Master: 2cb8830304e145db8591b8ff765eeef3 (127.0.98.3:45697)
I20260619 14:46:59.719199 7098 ts_manager.cc:194] Registered new tserver with Master: 2cb8830304e145db8591b8ff765eeef3 (127.0.98.3:45697)
I20260619 14:46:59.719210 6497 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53802:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:46:59.722532 6497 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:46:59.725211 7245 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a2a5f1050da4c09b1351fb6ecb3986a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4a2a5f1050da4c09b1351fb6ecb3986a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4a2a5f1050da4c09b1351fb6ecb3986a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37899 } } peers { permanent_uuid: "8e87b561972740938a4f82c45931705b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36857 } attrs { promote: false } } peers { permanent_uuid: "768cb9d7dfb349e3b12896639c2ddd4f" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 39275 } attrs { promote: true } } }
I20260619 14:46:59.725383 7245 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a2a5f1050da4c09b1351fb6ecb3986a [sys.catalog]: This master's current role is: LEADER
I20260619 14:46:59.728439 7752 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 4a2a5f1050da4c09b1351fb6ecb3986a: attempting to promote NON_VOTER 768cb9d7dfb349e3b12896639c2ddd4f to VOTER
I20260619 14:46:59.728876 7752 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a2a5f1050da4c09b1351fb6ecb3986a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4a2a5f1050da4c09b1351fb6ecb3986a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37899 } } peers { permanent_uuid: "8e87b561972740938a4f82c45931705b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36857 } attrs { promote: false } } peers { permanent_uuid: "768cb9d7dfb349e3b12896639c2ddd4f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39275 } attrs { promote: false } }
I20260619 14:46:59.730664 7108 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8e87b561972740938a4f82c45931705b [term 1 FOLLOWER]: Refusing update from remote peer 4a2a5f1050da4c09b1351fb6ecb3986a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260619 14:46:59.731045 7291 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 768cb9d7dfb349e3b12896639c2ddd4f [term 1 LEARNER]: Refusing update from remote peer 4a2a5f1050da4c09b1351fb6ecb3986a: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260619 14:46:59.731051 7751 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4a2a5f1050da4c09b1351fb6ecb3986a [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e87b561972740938a4f82c45931705b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36857 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260619 14:46:59.731805 7751 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4a2a5f1050da4c09b1351fb6ecb3986a [LEADER]: Connected to new peer: Peer: permanent_uuid: "768cb9d7dfb349e3b12896639c2ddd4f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39275 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260619 14:46:59.733574 7752 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4a2a5f1050da4c09b1351fb6ecb3986a [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 768cb9d7dfb349e3b12896639c2ddd4f (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4a2a5f1050da4c09b1351fb6ecb3986a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37899 } } peers { permanent_uuid: "8e87b561972740938a4f82c45931705b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36857 } attrs { promote: false } } peers { permanent_uuid: "768cb9d7dfb349e3b12896639c2ddd4f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39275 } attrs { promote: false } } }
I20260619 14:46:59.734318 7108 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8e87b561972740938a4f82c45931705b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 768cb9d7dfb349e3b12896639c2ddd4f (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4a2a5f1050da4c09b1351fb6ecb3986a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37899 } } peers { permanent_uuid: "8e87b561972740938a4f82c45931705b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36857 } attrs { promote: false } } peers { permanent_uuid: "768cb9d7dfb349e3b12896639c2ddd4f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39275 } attrs { promote: false } } }
I20260619 14:46:59.735184 7751 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a2a5f1050da4c09b1351fb6ecb3986a [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4a2a5f1050da4c09b1351fb6ecb3986a" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4a2a5f1050da4c09b1351fb6ecb3986a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37899 } } peers { permanent_uuid: "8e87b561972740938a4f82c45931705b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36857 } attrs { promote: false } } peers { permanent_uuid: "768cb9d7dfb349e3b12896639c2ddd4f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39275 } attrs { promote: false } } }
I20260619 14:46:59.735365 7751 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a2a5f1050da4c09b1351fb6ecb3986a [sys.catalog]: This master's current role is: LEADER
I20260619 14:46:59.735067 7291 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 768cb9d7dfb349e3b12896639c2ddd4f [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 768cb9d7dfb349e3b12896639c2ddd4f (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4a2a5f1050da4c09b1351fb6ecb3986a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37899 } } peers { permanent_uuid: "8e87b561972740938a4f82c45931705b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36857 } attrs { promote: false } } peers { permanent_uuid: "768cb9d7dfb349e3b12896639c2ddd4f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39275 } attrs { promote: false } } }
I20260619 14:46:59.735671 7752 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a2a5f1050da4c09b1351fb6ecb3986a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4a2a5f1050da4c09b1351fb6ecb3986a" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4a2a5f1050da4c09b1351fb6ecb3986a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37899 } } peers { permanent_uuid: "8e87b561972740938a4f82c45931705b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36857 } attrs { promote: false } } peers { permanent_uuid: "768cb9d7dfb349e3b12896639c2ddd4f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39275 } attrs { promote: false } } }
I20260619 14:46:59.735790 7752 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a2a5f1050da4c09b1351fb6ecb3986a [sys.catalog]: This master's current role is: LEADER
I20260619 14:46:59.735348 7245 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a2a5f1050da4c09b1351fb6ecb3986a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4a2a5f1050da4c09b1351fb6ecb3986a" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4a2a5f1050da4c09b1351fb6ecb3986a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37899 } } peers { permanent_uuid: "8e87b561972740938a4f82c45931705b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36857 } attrs { promote: false } } peers { permanent_uuid: "768cb9d7dfb349e3b12896639c2ddd4f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39275 } attrs { promote: false } } }
I20260619 14:46:59.735848 7245 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a2a5f1050da4c09b1351fb6ecb3986a [sys.catalog]: This master's current role is: LEADER
I20260619 14:46:59.737222 7323 sys_catalog.cc:455] T 00000000000000000000000000000000 P 768cb9d7dfb349e3b12896639c2ddd4f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4a2a5f1050da4c09b1351fb6ecb3986a" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4a2a5f1050da4c09b1351fb6ecb3986a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37899 } } peers { permanent_uuid: "8e87b561972740938a4f82c45931705b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36857 } attrs { promote: false } } peers { permanent_uuid: "768cb9d7dfb349e3b12896639c2ddd4f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39275 } attrs { promote: false } } }
I20260619 14:46:59.737398 7323 sys_catalog.cc:458] T 00000000000000000000000000000000 P 768cb9d7dfb349e3b12896639c2ddd4f [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:46:59.738725 7761 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e87b561972740938a4f82c45931705b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4a2a5f1050da4c09b1351fb6ecb3986a" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4a2a5f1050da4c09b1351fb6ecb3986a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37899 } } peers { permanent_uuid: "8e87b561972740938a4f82c45931705b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36857 } attrs { promote: false } } peers { permanent_uuid: "768cb9d7dfb349e3b12896639c2ddd4f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39275 } attrs { promote: false } } }
I20260619 14:46:59.738893 7761 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e87b561972740938a4f82c45931705b [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:46:59.748068 7054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:59.740444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37475 (local address 127.0.0.1:60610)
0619 14:46:59.740773 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:59.740794 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:59.740890 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:59.742668 (+ 1778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:59.742671 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:59.742691 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:46:59.742840 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:59.742850 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:59.743897 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:59.743900 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:59.745221 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:59.745232 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:59.747636 (+ 2404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:59.747641 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:59.747739 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:59.747769 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 14:46:59.747825 (+ 56us) client_negotiation.cc:240] Negotiation successful
0619 14:46:59.747905 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233}
I20260619 14:46:59.748536 7764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:59.741188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37385 (local address 127.0.0.1:54166)
0619 14:46:59.743790 (+ 2602us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:59.743814 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:59.743918 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:59.745218 (+ 1300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:59.745221 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:59.745239 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:46:59.745328 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:59.745334 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:59.746301 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:59.746306 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:59.747120 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:59.747129 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:59.748099 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:59.748101 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:59.748197 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:59.748230 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 14:46:59.748304 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 14:46:59.748375 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2534,"thread_start_us":85,"threads_started":1}
I20260619 14:46:59.748737 7763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:59.740753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45697 (local address 127.0.0.1:32940)
0619 14:46:59.741230 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:59.741246 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:59.741411 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:59.743514 (+ 2103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:59.743517 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:59.743537 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:46:59.743653 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:59.743663 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:59.746089 (+ 2426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:59.746096 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:59.746976 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:59.746986 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:59.748285 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:59.748290 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:59.748397 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:59.748422 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 14:46:59.748501 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 14:46:59.748581 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":116,"threads_started":1}
I20260619 14:46:59.749060 7766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:59.741559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54166 (local address 127.0.98.2:37385)
0619 14:46:59.741915 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:59.741932 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:59.744759 (+ 2827us) server_negotiation.cc:407] Connection header received
0619 14:46:59.744848 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:59.744852 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:59.745055 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:59.745250 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:46:59.745420 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:59.746187 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:59.747250 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:59.747796 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:59.747926 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:59.748046 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:59.748808 (+ 762us) server_negotiation.cc:299] Negotiation successful
0619 14:46:59.748928 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":90,"threads_started":1}
I20260619 14:46:59.749439 7765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:59.741290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32940 (local address 127.0.98.3:45697)
0619 14:46:59.743051 (+ 1761us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:59.743072 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:59.743100 (+ 28us) server_negotiation.cc:407] Connection header received
0619 14:46:59.743179 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:59.743201 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:59.743361 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:59.743543 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:46:59.744040 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:59.745970 (+ 1930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:59.747144 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:59.748066 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:59.748321 (+ 255us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:59.748499 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:59.748559 (+ 60us) server_negotiation.cc:299] Negotiation successful
0619 14:46:59.748706 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1637,"thread_start_us":128,"threads_started":1}
I20260619 14:46:59.749545 7762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:59.740699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60610 (local address 127.0.98.1:37475)
0619 14:46:59.742184 (+ 1485us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:59.742190 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:59.742211 (+ 21us) server_negotiation.cc:407] Connection header received
0619 14:46:59.742282 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:59.742286 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:59.742520 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:59.742705 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:46:59.743034 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:59.743784 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:59.745384 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:59.747483 (+ 2099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:59.749053 (+ 1570us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:59.749210 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:59.749264 (+ 54us) server_negotiation.cc:299] Negotiation successful
0619 14:46:59.749367 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1393,"thread_start_us":102,"threads_started":1}
I20260619 14:46:59.751812 7531 tablet_service.cc:1467] Processing CreateTablet for tablet cb9fa85eb4b84dd6ae6ae81920200151 (DEFAULT_TABLE table=test-table [id=96b2428bceb04502b3da431b642ac510]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:46:59.752302 7667 tablet_service.cc:1467] Processing CreateTablet for tablet cb9fa85eb4b84dd6ae6ae81920200151 (DEFAULT_TABLE table=test-table [id=96b2428bceb04502b3da431b642ac510]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:46:59.752956 7531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb9fa85eb4b84dd6ae6ae81920200151. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:46:59.752774 7394 tablet_service.cc:1467] Processing CreateTablet for tablet cb9fa85eb4b84dd6ae6ae81920200151 (DEFAULT_TABLE table=test-table [id=96b2428bceb04502b3da431b642ac510]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:46:59.753539 7667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb9fa85eb4b84dd6ae6ae81920200151. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:46:59.753805 7394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb9fa85eb4b84dd6ae6ae81920200151. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:46:59.762010 7767 tablet_bootstrap.cc:492] T cb9fa85eb4b84dd6ae6ae81920200151 P 0f513b290a8c49ee81a89fa8d2152c04: Bootstrap starting.
I20260619 14:46:59.763536 7769 tablet_bootstrap.cc:492] T cb9fa85eb4b84dd6ae6ae81920200151 P 1b116e7a13764f34b66cdbb5e53338e4: Bootstrap starting.
I20260619 14:46:59.764811 7767 tablet_bootstrap.cc:654] T cb9fa85eb4b84dd6ae6ae81920200151 P 0f513b290a8c49ee81a89fa8d2152c04: Neither blocks nor log segments found. Creating new log.
I20260619 14:46:59.765144 7768 tablet_bootstrap.cc:492] T cb9fa85eb4b84dd6ae6ae81920200151 P 2cb8830304e145db8591b8ff765eeef3: Bootstrap starting.
I20260619 14:46:59.765816 7767 log.cc:824] T cb9fa85eb4b84dd6ae6ae81920200151 P 0f513b290a8c49ee81a89fa8d2152c04: Log is configured to *not* fsync() on all Append() calls
I20260619 14:46:59.765957 7769 tablet_bootstrap.cc:654] T cb9fa85eb4b84dd6ae6ae81920200151 P 1b116e7a13764f34b66cdbb5e53338e4: Neither blocks nor log segments found. Creating new log.
I20260619 14:46:59.767002 7769 log.cc:824] T cb9fa85eb4b84dd6ae6ae81920200151 P 1b116e7a13764f34b66cdbb5e53338e4: Log is configured to *not* fsync() on all Append() calls
I20260619 14:46:59.767701 7768 tablet_bootstrap.cc:654] T cb9fa85eb4b84dd6ae6ae81920200151 P 2cb8830304e145db8591b8ff765eeef3: Neither blocks nor log segments found. Creating new log.
I20260619 14:46:59.768584 7767 tablet_bootstrap.cc:492] T cb9fa85eb4b84dd6ae6ae81920200151 P 0f513b290a8c49ee81a89fa8d2152c04: No bootstrap required, opened a new log
I20260619 14:46:59.768613 7768 log.cc:824] T cb9fa85eb4b84dd6ae6ae81920200151 P 2cb8830304e145db8591b8ff765eeef3: Log is configured to *not* fsync() on all Append() calls
I20260619 14:46:59.768860 7767 ts_tablet_manager.cc:1397] T cb9fa85eb4b84dd6ae6ae81920200151 P 0f513b290a8c49ee81a89fa8d2152c04: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260619 14:46:59.769703 7769 tablet_bootstrap.cc:492] T cb9fa85eb4b84dd6ae6ae81920200151 P 1b116e7a13764f34b66cdbb5e53338e4: No bootstrap required, opened a new log
I20260619 14:46:59.769904 7769 ts_tablet_manager.cc:1397] T cb9fa85eb4b84dd6ae6ae81920200151 P 1b116e7a13764f34b66cdbb5e53338e4: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260619 14:46:59.770604 7768 tablet_bootstrap.cc:492] T cb9fa85eb4b84dd6ae6ae81920200151 P 2cb8830304e145db8591b8ff765eeef3: No bootstrap required, opened a new log
I20260619 14:46:59.770833 7768 ts_tablet_manager.cc:1397] T cb9fa85eb4b84dd6ae6ae81920200151 P 2cb8830304e145db8591b8ff765eeef3: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 14:46:59.773533 7767 raft_consensus.cc:348] T cb9fa85eb4b84dd6ae6ae81920200151 P 0f513b290a8c49ee81a89fa8d2152c04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b116e7a13764f34b66cdbb5e53338e4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37475 } } peers { permanent_uuid: "2cb8830304e145db8591b8ff765eeef3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45697 } } peers { permanent_uuid: "0f513b290a8c49ee81a89fa8d2152c04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37385 } }
I20260619 14:46:59.773854 7767 raft_consensus.cc:374] T cb9fa85eb4b84dd6ae6ae81920200151 P 0f513b290a8c49ee81a89fa8d2152c04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:46:59.773924 7767 raft_consensus.cc:729] T cb9fa85eb4b84dd6ae6ae81920200151 P 0f513b290a8c49ee81a89fa8d2152c04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f513b290a8c49ee81a89fa8d2152c04, State: Initialized, Role: FOLLOWER
I20260619 14:46:59.774233 7769 raft_consensus.cc:348] T cb9fa85eb4b84dd6ae6ae81920200151 P 1b116e7a13764f34b66cdbb5e53338e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b116e7a13764f34b66cdbb5e53338e4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37475 } } peers { permanent_uuid: "2cb8830304e145db8591b8ff765eeef3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45697 } } peers { permanent_uuid: "0f513b290a8c49ee81a89fa8d2152c04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37385 } }
I20260619 14:46:59.774415 7767 consensus_queue.cc:260] T cb9fa85eb4b84dd6ae6ae81920200151 P 0f513b290a8c49ee81a89fa8d2152c04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b116e7a13764f34b66cdbb5e53338e4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37475 } } peers { permanent_uuid: "2cb8830304e145db8591b8ff765eeef3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45697 } } peers { permanent_uuid: "0f513b290a8c49ee81a89fa8d2152c04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37385 } }
I20260619 14:46:59.774518 7769 raft_consensus.cc:374] T cb9fa85eb4b84dd6ae6ae81920200151 P 1b116e7a13764f34b66cdbb5e53338e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:46:59.774588 7769 raft_consensus.cc:729] T cb9fa85eb4b84dd6ae6ae81920200151 P 1b116e7a13764f34b66cdbb5e53338e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b116e7a13764f34b66cdbb5e53338e4, State: Initialized, Role: FOLLOWER
I20260619 14:46:59.774837 7768 raft_consensus.cc:348] T cb9fa85eb4b84dd6ae6ae81920200151 P 2cb8830304e145db8591b8ff765eeef3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b116e7a13764f34b66cdbb5e53338e4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37475 } } peers { permanent_uuid: "2cb8830304e145db8591b8ff765eeef3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45697 } } peers { permanent_uuid: "0f513b290a8c49ee81a89fa8d2152c04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37385 } }
I20260619 14:46:59.775110 7768 raft_consensus.cc:374] T cb9fa85eb4b84dd6ae6ae81920200151 P 2cb8830304e145db8591b8ff765eeef3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:46:59.775069 7769 consensus_queue.cc:260] T cb9fa85eb4b84dd6ae6ae81920200151 P 1b116e7a13764f34b66cdbb5e53338e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b116e7a13764f34b66cdbb5e53338e4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37475 } } peers { permanent_uuid: "2cb8830304e145db8591b8ff765eeef3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45697 } } peers { permanent_uuid: "0f513b290a8c49ee81a89fa8d2152c04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37385 } }
I20260619 14:46:59.775187 7768 raft_consensus.cc:729] T cb9fa85eb4b84dd6ae6ae81920200151 P 2cb8830304e145db8591b8ff765eeef3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cb8830304e145db8591b8ff765eeef3, State: Initialized, Role: FOLLOWER
I20260619 14:46:59.775502 7597 heartbeater.cc:503] Master 127.0.98.62:37899 was elected leader, sending a full tablet report...
I20260619 14:46:59.775552 7767 ts_tablet_manager.cc:1428] T cb9fa85eb4b84dd6ae6ae81920200151 P 0f513b290a8c49ee81a89fa8d2152c04: Time spent starting tablet: real 0.007s user 0.004s sys 0.003s
I20260619 14:46:59.775667 7768 consensus_queue.cc:260] T cb9fa85eb4b84dd6ae6ae81920200151 P 2cb8830304e145db8591b8ff765eeef3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b116e7a13764f34b66cdbb5e53338e4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37475 } } peers { permanent_uuid: "2cb8830304e145db8591b8ff765eeef3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45697 } } peers { permanent_uuid: "0f513b290a8c49ee81a89fa8d2152c04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37385 } }
I20260619 14:46:59.776257 7769 ts_tablet_manager.cc:1428] T cb9fa85eb4b84dd6ae6ae81920200151 P 1b116e7a13764f34b66cdbb5e53338e4: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260619 14:46:59.776707 7768 ts_tablet_manager.cc:1428] T cb9fa85eb4b84dd6ae6ae81920200151 P 2cb8830304e145db8591b8ff765eeef3: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260619 14:46:59.776782 7733 heartbeater.cc:503] Master 127.0.98.62:37899 was elected leader, sending a full tablet report...
I20260619 14:46:59.777027 7460 heartbeater.cc:503] Master 127.0.98.62:37899 was elected leader, sending a full tablet report...
I20260619 14:46:59.869575 7773 raft_consensus.cc:482] T cb9fa85eb4b84dd6ae6ae81920200151 P 0f513b290a8c49ee81a89fa8d2152c04 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:46:59.869884 7773 raft_consensus.cc:504] T cb9fa85eb4b84dd6ae6ae81920200151 P 0f513b290a8c49ee81a89fa8d2152c04 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b116e7a13764f34b66cdbb5e53338e4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37475 } } peers { permanent_uuid: "2cb8830304e145db8591b8ff765eeef3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45697 } } peers { permanent_uuid: "0f513b290a8c49ee81a89fa8d2152c04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37385 } }
I20260619 14:46:59.871250 7773 leader_election.cc:288] T cb9fa85eb4b84dd6ae6ae81920200151 P 0f513b290a8c49ee81a89fa8d2152c04 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b116e7a13764f34b66cdbb5e53338e4 (127.0.98.1:37475), 2cb8830304e145db8591b8ff765eeef3 (127.0.98.3:45697)
I20260619 14:46:59.875525 7762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:59.871749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52051 (local address 127.0.98.1:37475)
0619 14:46:59.871877 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:59.871881 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:59.871897 (+ 16us) server_negotiation.cc:407] Connection header received
0619 14:46:59.872059 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:59.872062 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:59.872128 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:59.872216 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:46:59.872505 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:59.873107 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:59.874159 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:59.874908 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:59.875044 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:59.875111 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:59.875307 (+ 196us) server_negotiation.cc:299] Negotiation successful
0619 14:46:59.875376 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 14:46:59.875531 7765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:59.871864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37283 (local address 127.0.98.3:45697)
0619 14:46:59.872017 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:59.872023 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:59.872039 (+ 16us) server_negotiation.cc:407] Connection header received
0619 14:46:59.872182 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:59.872185 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:59.872234 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:59.872321 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:46:59.872923 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:59.873483 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:59.874456 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:59.874953 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:59.875098 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:59.875160 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:59.875336 (+ 176us) server_negotiation.cc:299] Negotiation successful
0619 14:46:59.875409 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 14:46:59.875546 7601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:59.871592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37475 (local address 127.0.98.2:52051)
0619 14:46:59.871783 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:59.871801 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:59.871916 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:59.872227 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:59.872230 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:59.872248 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:46:59.872377 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:59.872382 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:59.873223 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:59.873228 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:59.874024 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:59.874032 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:59.875060 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:59.875064 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:59.875179 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:59.875200 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 14:46:59.875292 (+ 92us) client_negotiation.cc:240] Negotiation successful
0619 14:46:59.875375 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260619 14:46:59.875551 7602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:59.871663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45697 (local address 127.0.98.2:37283)
0619 14:46:59.871956 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0619 14:46:59.871971 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:46:59.872058 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:46:59.872332 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:46:59.872335 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:46:59.872350 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:46:59.872804 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:59.872812 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:59.873608 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:59.873613 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:59.874329 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:46:59.874337 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:46:59.875099 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:46:59.875103 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:46:59.875213 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:59.875231 (+ 18us) client_negotiation.cc:769] Sending connection context
0619 14:46:59.875321 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 14:46:59.875391 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"mutex_wait_us":121}
I20260619 14:46:59.876354 7414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb9fa85eb4b84dd6ae6ae81920200151" candidate_uuid: "0f513b290a8c49ee81a89fa8d2152c04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b116e7a13764f34b66cdbb5e53338e4" is_pre_election: true
I20260619 14:46:59.876516 7687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb9fa85eb4b84dd6ae6ae81920200151" candidate_uuid: "0f513b290a8c49ee81a89fa8d2152c04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cb8830304e145db8591b8ff765eeef3" is_pre_election: true
I20260619 14:46:59.876859 7687 raft_consensus.cc:2454] T cb9fa85eb4b84dd6ae6ae81920200151 P 2cb8830304e145db8591b8ff765eeef3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f513b290a8c49ee81a89fa8d2152c04 in term 0.
I20260619 14:46:59.876860 7414 raft_consensus.cc:2454] T cb9fa85eb4b84dd6ae6ae81920200151 P 1b116e7a13764f34b66cdbb5e53338e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f513b290a8c49ee81a89fa8d2152c04 in term 0.
I20260619 14:46:59.877386 7487 leader_election.cc:302] T cb9fa85eb4b84dd6ae6ae81920200151 P 0f513b290a8c49ee81a89fa8d2152c04 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0f513b290a8c49ee81a89fa8d2152c04, 1b116e7a13764f34b66cdbb5e53338e4, 2cb8830304e145db8591b8ff765eeef3; no voters:
I20260619 14:46:59.877724 7773 raft_consensus.cc:2781] T cb9fa85eb4b84dd6ae6ae81920200151 P 0f513b290a8c49ee81a89fa8d2152c04 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:46:59.877822 7773 raft_consensus.cc:482] T cb9fa85eb4b84dd6ae6ae81920200151 P 0f513b290a8c49ee81a89fa8d2152c04 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:46:59.877866 7773 raft_consensus.cc:3037] T cb9fa85eb4b84dd6ae6ae81920200151 P 0f513b290a8c49ee81a89fa8d2152c04 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:46:59.878943 7773 raft_consensus.cc:504] T cb9fa85eb4b84dd6ae6ae81920200151 P 0f513b290a8c49ee81a89fa8d2152c04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b116e7a13764f34b66cdbb5e53338e4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37475 } } peers { permanent_uuid: "2cb8830304e145db8591b8ff765eeef3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45697 } } peers { permanent_uuid: "0f513b290a8c49ee81a89fa8d2152c04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37385 } }
I20260619 14:46:59.879356 7773 leader_election.cc:288] T cb9fa85eb4b84dd6ae6ae81920200151 P 0f513b290a8c49ee81a89fa8d2152c04 [CANDIDATE]: Term 1 election: Requested vote from peers 1b116e7a13764f34b66cdbb5e53338e4 (127.0.98.1:37475), 2cb8830304e145db8591b8ff765eeef3 (127.0.98.3:45697)
I20260619 14:46:59.879799 7414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb9fa85eb4b84dd6ae6ae81920200151" candidate_uuid: "0f513b290a8c49ee81a89fa8d2152c04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b116e7a13764f34b66cdbb5e53338e4"
I20260619 14:46:59.879987 7414 raft_consensus.cc:3037] T cb9fa85eb4b84dd6ae6ae81920200151 P 1b116e7a13764f34b66cdbb5e53338e4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:46:59.879981 7687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb9fa85eb4b84dd6ae6ae81920200151" candidate_uuid: "0f513b290a8c49ee81a89fa8d2152c04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cb8830304e145db8591b8ff765eeef3"
I20260619 14:46:59.880149 7687 raft_consensus.cc:3037] T cb9fa85eb4b84dd6ae6ae81920200151 P 2cb8830304e145db8591b8ff765eeef3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:46:59.881386 7687 raft_consensus.cc:2454] T cb9fa85eb4b84dd6ae6ae81920200151 P 2cb8830304e145db8591b8ff765eeef3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f513b290a8c49ee81a89fa8d2152c04 in term 1.
I20260619 14:46:59.881430 7414 raft_consensus.cc:2454] T cb9fa85eb4b84dd6ae6ae81920200151 P 1b116e7a13764f34b66cdbb5e53338e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f513b290a8c49ee81a89fa8d2152c04 in term 1.
I20260619 14:46:59.881798 7484 leader_election.cc:302] T cb9fa85eb4b84dd6ae6ae81920200151 P 0f513b290a8c49ee81a89fa8d2152c04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f513b290a8c49ee81a89fa8d2152c04, 2cb8830304e145db8591b8ff765eeef3; no voters:
I20260619 14:46:59.882097 7773 raft_consensus.cc:2781] T cb9fa85eb4b84dd6ae6ae81920200151 P 0f513b290a8c49ee81a89fa8d2152c04 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:46:59.882369 7773 raft_consensus.cc:686] T cb9fa85eb4b84dd6ae6ae81920200151 P 0f513b290a8c49ee81a89fa8d2152c04 [term 1 LEADER]: Becoming Leader. State: Replica: 0f513b290a8c49ee81a89fa8d2152c04, State: Running, Role: LEADER
I20260619 14:46:59.882804 7773 consensus_queue.cc:237] T cb9fa85eb4b84dd6ae6ae81920200151 P 0f513b290a8c49ee81a89fa8d2152c04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b116e7a13764f34b66cdbb5e53338e4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37475 } } peers { permanent_uuid: "2cb8830304e145db8591b8ff765eeef3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45697 } } peers { permanent_uuid: "0f513b290a8c49ee81a89fa8d2152c04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37385 } }
I20260619 14:46:59.886495 6496 catalog_manager.cc:5495] T cb9fa85eb4b84dd6ae6ae81920200151 P 0f513b290a8c49ee81a89fa8d2152c04 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0f513b290a8c49ee81a89fa8d2152c04 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "0f513b290a8c49ee81a89fa8d2152c04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b116e7a13764f34b66cdbb5e53338e4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2cb8830304e145db8591b8ff765eeef3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f513b290a8c49ee81a89fa8d2152c04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37385 } health_report { overall_health: HEALTHY } } }
I20260619 14:46:59.938531 7766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:59.934263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54174 (local address 127.0.98.2:37385)
0619 14:46:59.934417 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:59.934533 (+ 116us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:59.934550 (+ 17us) server_negotiation.cc:407] Connection header received
0619 14:46:59.934628 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:59.934630 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:59.934691 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:59.934839 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:46:59.935111 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:59.935711 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:59.936876 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:59.937123 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:59.937248 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:46:59.937608 (+ 360us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:46:59.938073 (+ 465us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:46:59.938222 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 14:46:59.938283 (+ 61us) server_negotiation.cc:299] Negotiation successful
0619 14:46:59.938349 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260619 14:46:59.943295 7736 tablet.cc:2369] T cb9fa85eb4b84dd6ae6ae81920200151 P 2cb8830304e145db8591b8ff765eeef3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:46:59.945658 7687 raft_consensus.cc:1261] T cb9fa85eb4b84dd6ae6ae81920200151 P 2cb8830304e145db8591b8ff765eeef3 [term 1 FOLLOWER]: Refusing update from remote peer 0f513b290a8c49ee81a89fa8d2152c04: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:46:59.945659 7414 raft_consensus.cc:1261] T cb9fa85eb4b84dd6ae6ae81920200151 P 1b116e7a13764f34b66cdbb5e53338e4 [term 1 FOLLOWER]: Refusing update from remote peer 0f513b290a8c49ee81a89fa8d2152c04: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:46:59.946461 7773 consensus_queue.cc:1035] T cb9fa85eb4b84dd6ae6ae81920200151 P 0f513b290a8c49ee81a89fa8d2152c04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b116e7a13764f34b66cdbb5e53338e4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:46:59.946707 7776 consensus_queue.cc:1035] T cb9fa85eb4b84dd6ae6ae81920200151 P 0f513b290a8c49ee81a89fa8d2152c04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cb8830304e145db8591b8ff765eeef3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:46:59.959283 7781 mvcc.cc:205] Tried to move back new op lower bound from 7298582200089395200 to 7298582199842267136. Current Snapshot: MvccSnapshot[applied={T|T < 7298582200089395200}]
I20260619 14:46:59.969890 6496 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53802:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:46:59.970078 6496 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53802:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:46:59.976751 6496 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4a2a5f1050da4c09b1351fb6ecb3986a: Sending DeleteTablet for 3 replicas of tablet cb9fa85eb4b84dd6ae6ae81920200151
I20260619 14:46:59.977892 7394 tablet_service.cc:1514] Processing DeleteTablet for tablet cb9fa85eb4b84dd6ae6ae81920200151 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:46:59 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60610
I20260619 14:46:59.978063 7667 tablet_service.cc:1514] Processing DeleteTablet for tablet cb9fa85eb4b84dd6ae6ae81920200151 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:46:59 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:32940
I20260619 14:46:59.979213 7531 tablet_service.cc:1514] Processing DeleteTablet for tablet cb9fa85eb4b84dd6ae6ae81920200151 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:46:59 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54166
I20260619 14:46:59.979799 7791 tablet_replica.cc:330] T cb9fa85eb4b84dd6ae6ae81920200151 P 0f513b290a8c49ee81a89fa8d2152c04: stopping tablet replica
I20260619 14:46:59.980203 7791 raft_consensus.cc:2229] T cb9fa85eb4b84dd6ae6ae81920200151 P 0f513b290a8c49ee81a89fa8d2152c04 [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:46:59.980755 7791 raft_consensus.cc:2258] T cb9fa85eb4b84dd6ae6ae81920200151 P 0f513b290a8c49ee81a89fa8d2152c04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:46:59.982242 7789 tablet_replica.cc:330] T cb9fa85eb4b84dd6ae6ae81920200151 P 2cb8830304e145db8591b8ff765eeef3: stopping tablet replica
I20260619 14:46:59.982497 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 6465
I20260619 14:46:59.983963 7790 tablet_replica.cc:330] T cb9fa85eb4b84dd6ae6ae81920200151 P 1b116e7a13764f34b66cdbb5e53338e4: stopping tablet replica
I20260619 14:46:59.984313 7790 raft_consensus.cc:2229] T cb9fa85eb4b84dd6ae6ae81920200151 P 1b116e7a13764f34b66cdbb5e53338e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:46:59.984328 7791 ts_tablet_manager.cc:1905] T cb9fa85eb4b84dd6ae6ae81920200151 P 0f513b290a8c49ee81a89fa8d2152c04: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:46:59.984709 7790 raft_consensus.cc:2258] T cb9fa85eb4b84dd6ae6ae81920200151 P 1b116e7a13764f34b66cdbb5e53338e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:46:59.982571 7789 raft_consensus.cc:2229] T cb9fa85eb4b84dd6ae6ae81920200151 P 2cb8830304e145db8591b8ff765eeef3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:46:59.985204 7789 raft_consensus.cc:2258] T cb9fa85eb4b84dd6ae6ae81920200151 P 2cb8830304e145db8591b8ff765eeef3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:46:59.987079 7789 ts_tablet_manager.cc:1905] T cb9fa85eb4b84dd6ae6ae81920200151 P 2cb8830304e145db8591b8ff765eeef3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:46:59.988065 7791 ts_tablet_manager.cc:1918] T cb9fa85eb4b84dd6ae6ae81920200151 P 0f513b290a8c49ee81a89fa8d2152c04: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:46:59.988238 7791 log.cc:1196] T cb9fa85eb4b84dd6ae6ae81920200151 P 0f513b290a8c49ee81a89fa8d2152c04: Deleting WAL directory at /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/ts-1/wal/wals/cb9fa85eb4b84dd6ae6ae81920200151
I20260619 14:46:59.988750 7791 ts_tablet_manager.cc:1939] T cb9fa85eb4b84dd6ae6ae81920200151 P 0f513b290a8c49ee81a89fa8d2152c04: Deleting consensus metadata
I20260619 14:46:59.990437 7790 ts_tablet_manager.cc:1905] T cb9fa85eb4b84dd6ae6ae81920200151 P 1b116e7a13764f34b66cdbb5e53338e4: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260619 14:46:59.991698 7485 connection.cc:535] server connection from 127.0.0.1:54166 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 14:46:59.992252 7789 ts_tablet_manager.cc:1918] T cb9fa85eb4b84dd6ae6ae81920200151 P 2cb8830304e145db8591b8ff765eeef3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:46:59.992425 7789 log.cc:1196] T cb9fa85eb4b84dd6ae6ae81920200151 P 2cb8830304e145db8591b8ff765eeef3: Deleting WAL directory at /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/ts-2/wal/wals/cb9fa85eb4b84dd6ae6ae81920200151
I20260619 14:46:59.993228 7789 ts_tablet_manager.cc:1939] T cb9fa85eb4b84dd6ae6ae81920200151 P 2cb8830304e145db8591b8ff765eeef3: Deleting consensus metadata
I20260619 14:46:59.993667 7790 ts_tablet_manager.cc:1918] T cb9fa85eb4b84dd6ae6ae81920200151 P 1b116e7a13764f34b66cdbb5e53338e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:46:59.993778 7790 log.cc:1196] T cb9fa85eb4b84dd6ae6ae81920200151 P 1b116e7a13764f34b66cdbb5e53338e4: Deleting WAL directory at /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/ts-0/wal/wals/cb9fa85eb4b84dd6ae6ae81920200151
W20260619 14:46:59.995062 7623 connection.cc:441] server connection from 127.0.0.1:32940 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:32940 (request call id 1) could send its response
I20260619 14:46:59.997256 7790 ts_tablet_manager.cc:1939] T cb9fa85eb4b84dd6ae6ae81920200151 P 1b116e7a13764f34b66cdbb5e53338e4: Deleting consensus metadata
W20260619 14:46:59.998896 7348 connection.cc:441] server connection from 127.0.0.1:60610 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60610 (request call id 1) could send its response
I20260619 14:47:00.015252 7467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:59.997660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34586 (local address 127.0.98.60:39275)
0619 14:46:59.997774 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:59.997779 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:00.000259 (+ 2480us) server_negotiation.cc:407] Connection header received
0619 14:47:00.000387 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:00.000390 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:00.000452 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:00.000557 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:00.001637 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:00.002436 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:00.003397 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:00.003754 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:00.003886 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:00.011189 (+ 7303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:47:00.011193 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:47:00.011196 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:47:00.011230 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:47:00.013226 (+ 1996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:00.013793 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:00.013797 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:00.013799 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:00.013889 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:00.014190 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:00.014194 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:00.014195 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:00.014426 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:47:00.014549 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:00.014940 (+ 391us) server_negotiation.cc:299] Negotiation successful
0619 14:47:00.015079 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 14:47:00.016692 7220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:59.980847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57952 (local address 127.0.98.61:36857)
0619 14:46:59.980970 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:59.980975 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:59.980988 (+ 13us) server_negotiation.cc:407] Connection header received
0619 14:46:59.981029 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:59.981035 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:59.981087 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:59.981160 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:59.982082 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:59.982807 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:59.995899 (+ 13092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:46:59.996203 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:46:59.996348 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:00.010973 (+ 14625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:47:00.010979 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:47:00.010985 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:47:00.011043 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:47:00.013395 (+ 2352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:00.013962 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:00.013967 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:00.013972 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:00.014211 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:00.014969 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:00.014974 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:00.014976 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:00.015287 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:47:00.015422 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:00.016271 (+ 849us) server_negotiation.cc:299] Negotiation successful
0619 14:47:00.016445 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43,"spinlock_wait_cycles":1408}
I20260619 14:47:00.016697 7746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:59.996638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57956 (local address 127.0.98.61:36857)
0619 14:46:59.996795 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:59.996800 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:46:59.996996 (+ 196us) server_negotiation.cc:407] Connection header received
0619 14:46:59.997215 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:46:59.997221 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:46:59.997286 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:46:59.997343 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:46:59.999182 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:00.000100 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:00.001654 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:00.001935 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:00.002073 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:00.011758 (+ 9685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:47:00.011763 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:47:00.011769 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:47:00.011804 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:47:00.014011 (+ 2207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:00.014975 (+ 964us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:00.014978 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:00.014980 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:00.015135 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:00.015461 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:00.015465 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:00.015467 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:00.015671 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:47:00.015811 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:00.016404 (+ 593us) server_negotiation.cc:299] Negotiation successful
0619 14:47:00.016526 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38,"spinlock_wait_cycles":2816}
I20260619 14:47:00.026422 7748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:46:59.980575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34582 (local address 127.0.98.60:39275)
0619 14:46:59.980723 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 14:46:59.980729 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:00.017244 (+ 36515us) server_negotiation.cc:407] Connection header received
0619 14:47:00.017485 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:00.017488 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:00.017544 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:00.017603 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:00.018701 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:00.019310 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:00.020223 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:00.020517 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:00.020650 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:00.022307 (+ 1657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:47:00.022312 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:47:00.022315 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:47:00.022345 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:47:00.024069 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:00.024890 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:00.024894 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:00.024896 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:00.025009 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:00.025360 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:00.025364 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:00.025366 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:00.025596 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:47:00.025767 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:00.026074 (+ 307us) server_negotiation.cc:299] Negotiation successful
0619 14:47:00.026223 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260619 14:47:00.962360 7800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:00.961843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37899 (local address 127.0.98.3:33697)
0619 14:47:00.962111 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:00.962208 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":70,"threads_started":1}
W20260619 14:47:00.962865 7733 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37899 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:37899: connect: Connection refused (error 111)
I20260619 14:47:00.967794 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:00.967245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37899 (local address 127.0.98.2:33767)
0619 14:47:00.967537 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:00.967637 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":70,"threads_started":1}
I20260619 14:47:00.968042 7802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:00.967242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37899 (local address 127.0.98.1:52091)
0619 14:47:00.967545 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:00.967747 (+ 202us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":126,"threads_started":1}
W20260619 14:47:00.968374 7597 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37899 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:37899: connect: Connection refused (error 111)
W20260619 14:47:00.968379 7460 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37899 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:37899: connect: Connection refused (error 111)
I20260619 14:47:01.478084 7803 raft_consensus.cc:482] T 00000000000000000000000000000000 P 768cb9d7dfb349e3b12896639c2ddd4f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a2a5f1050da4c09b1351fb6ecb3986a)
I20260619 14:47:01.478327 7803 raft_consensus.cc:504] T 00000000000000000000000000000000 P 768cb9d7dfb349e3b12896639c2ddd4f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4a2a5f1050da4c09b1351fb6ecb3986a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37899 } } peers { permanent_uuid: "8e87b561972740938a4f82c45931705b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36857 } attrs { promote: false } } peers { permanent_uuid: "768cb9d7dfb349e3b12896639c2ddd4f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39275 } attrs { promote: false } }
I20260619 14:47:01.478946 7804 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8e87b561972740938a4f82c45931705b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4a2a5f1050da4c09b1351fb6ecb3986a)
I20260619 14:47:01.479149 7804 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8e87b561972740938a4f82c45931705b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4a2a5f1050da4c09b1351fb6ecb3986a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37899 } } peers { permanent_uuid: "8e87b561972740938a4f82c45931705b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36857 } attrs { promote: false } } peers { permanent_uuid: "768cb9d7dfb349e3b12896639c2ddd4f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39275 } attrs { promote: false } }
I20260619 14:47:01.479468 7803 leader_election.cc:288] T 00000000000000000000000000000000 P 768cb9d7dfb349e3b12896639c2ddd4f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a2a5f1050da4c09b1351fb6ecb3986a (127.0.98.62:37899), 8e87b561972740938a4f82c45931705b (127.0.98.61:36857)
I20260619 14:47:01.480204 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:01.479681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37899 (local address 127.0.0.1:54262)
0619 14:47:01.479984 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:01.480076 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":70,"threads_started":1}
I20260619 14:47:01.480365 7804 leader_election.cc:288] T 00000000000000000000000000000000 P 8e87b561972740938a4f82c45931705b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4a2a5f1050da4c09b1351fb6ecb3986a (127.0.98.62:37899), 768cb9d7dfb349e3b12896639c2ddd4f (127.0.98.60:39275)
W20260619 14:47:01.480427 7267 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37899: connect: Connection refused (error 111)
I20260619 14:47:01.480890 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:01.480443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37899 (local address 127.0.0.1:54272)
0619 14:47:01.480685 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:01.480768 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":68,"threads_started":1}
W20260619 14:47:01.481066 7084 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37899: connect: Connection refused (error 111)
I20260619 14:47:01.481701 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:01.481386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37899 (local address 127.0.0.1:54276)
0619 14:47:01.481562 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:01.481633 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
W20260619 14:47:01.481822 7267 leader_election.cc:334] T 00000000000000000000000000000000 P 768cb9d7dfb349e3b12896639c2ddd4f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a2a5f1050da4c09b1351fb6ecb3986a (127.0.98.62:37899): Network error: Client connection negotiation failed: client connection to 127.0.98.62:37899: connect: Connection refused (error 111)
I20260619 14:47:01.482244 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:01.481990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37899 (local address 127.0.0.1:54278)
0619 14:47:01.482113 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:01.482169 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260619 14:47:01.482405 7084 leader_election.cc:334] T 00000000000000000000000000000000 P 8e87b561972740938a4f82c45931705b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4a2a5f1050da4c09b1351fb6ecb3986a (127.0.98.62:37899): Network error: Client connection negotiation failed: client connection to 127.0.98.62:37899: connect: Connection refused (error 111)
I20260619 14:47:01.484298 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:01.480159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57964 (local address 127.0.98.61:36857)
0619 14:47:01.480538 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:01.480543 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:01.480557 (+ 14us) server_negotiation.cc:407] Connection header received
0619 14:47:01.480604 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:01.480608 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:01.480696 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:01.480787 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:47:01.481202 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:01.482044 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:01.483265 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:01.483746 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:01.483868 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:01.483927 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:01.484114 (+ 187us) server_negotiation.cc:299] Negotiation successful
0619 14:47:01.484183 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":32,"server-negotiator.queue_time_us":299,"thread_start_us":77,"threads_started":1}
I20260619 14:47:01.484313 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:01.479764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36857 (local address 127.0.0.1:57964)
0619 14:47:01.480025 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:01.480038 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:01.480147 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:01.480924 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:01.480929 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:01.480955 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:47:01.481057 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:01.481063 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:01.482159 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:01.482164 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:01.483138 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:01.483147 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:01.483868 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:01.483872 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:01.483961 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:01.483990 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 14:47:01.484090 (+ 100us) client_negotiation.cc:240] Negotiation successful
0619 14:47:01.484167 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":83,"threads_started":1}
I20260619 14:47:01.484807 7108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "768cb9d7dfb349e3b12896639c2ddd4f" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8e87b561972740938a4f82c45931705b" is_pre_election: true
I20260619 14:47:01.485020 7108 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8e87b561972740938a4f82c45931705b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 768cb9d7dfb349e3b12896639c2ddd4f in term 1.
I20260619 14:47:01.485435 7268 leader_election.cc:302] T 00000000000000000000000000000000 P 768cb9d7dfb349e3b12896639c2ddd4f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768cb9d7dfb349e3b12896639c2ddd4f, 8e87b561972740938a4f82c45931705b; no voters: 4a2a5f1050da4c09b1351fb6ecb3986a
I20260619 14:47:01.485601 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:01.480940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39275 (local address 127.0.0.1:34596)
0619 14:47:01.481049 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:01.481063 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:01.481159 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:01.482098 (+ 939us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:01.482102 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:01.482123 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:47:01.482222 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:01.482228 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:01.483175 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:01.483182 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:01.484133 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:01.484142 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:01.485143 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:01.485147 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:01.485255 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:01.485286 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 14:47:01.485385 (+ 99us) client_negotiation.cc:240] Negotiation successful
0619 14:47:01.485461 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":33,"thread_start_us":73,"threads_started":1}
I20260619 14:47:01.485615 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:01.481493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34596 (local address 127.0.98.60:39275)
0619 14:47:01.481803 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:01.481819 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:01.481836 (+ 17us) server_negotiation.cc:407] Connection header received
0619 14:47:01.481885 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:01.481888 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:01.481936 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:01.482337 (+ 401us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:47:01.482443 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:01.483049 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:01.484283 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:01.485017 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:01.485143 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:01.485200 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:01.485400 (+ 200us) server_negotiation.cc:299] Negotiation successful
0619 14:47:01.485483 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":175,"threads_started":1}
I20260619 14:47:01.485669 7803 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 768cb9d7dfb349e3b12896639c2ddd4f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260619 14:47:01.485721 7803 raft_consensus.cc:482] T 00000000000000000000000000000000 P 768cb9d7dfb349e3b12896639c2ddd4f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4a2a5f1050da4c09b1351fb6ecb3986a)
I20260619 14:47:01.485760 7803 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 768cb9d7dfb349e3b12896639c2ddd4f [term 1 FOLLOWER]: Advancing to term 2
I20260619 14:47:01.486080 7291 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8e87b561972740938a4f82c45931705b" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "768cb9d7dfb349e3b12896639c2ddd4f" is_pre_election: true
I20260619 14:47:01.486923 7803 raft_consensus.cc:504] T 00000000000000000000000000000000 P 768cb9d7dfb349e3b12896639c2ddd4f [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4a2a5f1050da4c09b1351fb6ecb3986a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37899 } } peers { permanent_uuid: "8e87b561972740938a4f82c45931705b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36857 } attrs { promote: false } } peers { permanent_uuid: "768cb9d7dfb349e3b12896639c2ddd4f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39275 } attrs { promote: false } }
I20260619 14:47:01.487162 7291 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 768cb9d7dfb349e3b12896639c2ddd4f [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8e87b561972740938a4f82c45931705b in current term 2: Already voted for candidate 768cb9d7dfb349e3b12896639c2ddd4f in this term.
I20260619 14:47:01.487372 7803 leader_election.cc:288] T 00000000000000000000000000000000 P 768cb9d7dfb349e3b12896639c2ddd4f [CANDIDATE]: Term 2 election: Requested vote from peers 4a2a5f1050da4c09b1351fb6ecb3986a (127.0.98.62:37899), 8e87b561972740938a4f82c45931705b (127.0.98.61:36857)
I20260619 14:47:01.487818 7108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "768cb9d7dfb349e3b12896639c2ddd4f" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8e87b561972740938a4f82c45931705b"
I20260619 14:47:01.487944 7108 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8e87b561972740938a4f82c45931705b [term 1 FOLLOWER]: Advancing to term 2
I20260619 14:47:01.488027 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:01.487728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37899 (local address 127.0.0.1:54294)
0619 14:47:01.487889 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:01.487948 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260619 14:47:01.488082 7083 leader_election.cc:302] T 00000000000000000000000000000000 P 8e87b561972740938a4f82c45931705b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8e87b561972740938a4f82c45931705b; no voters: 4a2a5f1050da4c09b1351fb6ecb3986a, 768cb9d7dfb349e3b12896639c2ddd4f
I20260619 14:47:01.488921 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:01.488715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37899 (local address 127.0.0.1:54300)
0619 14:47:01.488806 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:01.488858 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260619 14:47:01.489087 7267 leader_election.cc:334] T 00000000000000000000000000000000 P 768cb9d7dfb349e3b12896639c2ddd4f [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4a2a5f1050da4c09b1351fb6ecb3986a (127.0.98.62:37899): Network error: Client connection negotiation failed: client connection to 127.0.98.62:37899: connect: Connection refused (error 111)
I20260619 14:47:01.489439 7108 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8e87b561972740938a4f82c45931705b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 768cb9d7dfb349e3b12896639c2ddd4f in term 2.
I20260619 14:47:01.489569 7804 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 8e87b561972740938a4f82c45931705b [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 14:47:01.489879 7268 leader_election.cc:302] T 00000000000000000000000000000000 P 768cb9d7dfb349e3b12896639c2ddd4f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 768cb9d7dfb349e3b12896639c2ddd4f, 8e87b561972740938a4f82c45931705b; no voters: 4a2a5f1050da4c09b1351fb6ecb3986a
I20260619 14:47:01.490118 7803 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 768cb9d7dfb349e3b12896639c2ddd4f [term 2 FOLLOWER]: Leader election won for term 2
I20260619 14:47:01.490355 7803 raft_consensus.cc:686] T 00000000000000000000000000000000 P 768cb9d7dfb349e3b12896639c2ddd4f [term 2 LEADER]: Becoming Leader. State: Replica: 768cb9d7dfb349e3b12896639c2ddd4f, State: Running, Role: LEADER
I20260619 14:47:01.490648 7803 consensus_queue.cc:237] T 00000000000000000000000000000000 P 768cb9d7dfb349e3b12896639c2ddd4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4a2a5f1050da4c09b1351fb6ecb3986a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37899 } } peers { permanent_uuid: "8e87b561972740938a4f82c45931705b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36857 } attrs { promote: false } } peers { permanent_uuid: "768cb9d7dfb349e3b12896639c2ddd4f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39275 } attrs { promote: false } }
I20260619 14:47:01.491856 7813 sys_catalog.cc:455] T 00000000000000000000000000000000 P 768cb9d7dfb349e3b12896639c2ddd4f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 768cb9d7dfb349e3b12896639c2ddd4f. Latest consensus state: current_term: 2 leader_uuid: "768cb9d7dfb349e3b12896639c2ddd4f" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4a2a5f1050da4c09b1351fb6ecb3986a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37899 } } peers { permanent_uuid: "8e87b561972740938a4f82c45931705b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36857 } attrs { promote: false } } peers { permanent_uuid: "768cb9d7dfb349e3b12896639c2ddd4f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39275 } attrs { promote: false } } }
I20260619 14:47:01.492010 7813 sys_catalog.cc:458] T 00000000000000000000000000000000 P 768cb9d7dfb349e3b12896639c2ddd4f [sys.catalog]: This master's current role is: LEADER
I20260619 14:47:01.492403 7815 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:47:01.494874 7815 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:47:01.495306 7815 catalog_manager.cc:1259] Loaded cluster ID: 54bc711fead8420ebecba113ea139541
I20260619 14:47:01.495375 7815 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:47:01.495913 7815 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:47:01.496347 7815 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 768cb9d7dfb349e3b12896639c2ddd4f: Loaded TSK: 0
I20260619 14:47:01.497124 7815 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:47:01.553385 7279 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34586:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:47:01.556829 7279 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:47:01.560088 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:01.559738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37899 (local address 127.0.0.1:54318)
0619 14:47:01.559875 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:01.559999 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260619 14:47:01.560086 7108 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8e87b561972740938a4f82c45931705b [term 2 FOLLOWER]: Refusing update from remote peer 768cb9d7dfb349e3b12896639c2ddd4f: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260619 14:47:01.560708 7813 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 768cb9d7dfb349e3b12896639c2ddd4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e87b561972740938a4f82c45931705b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36857 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260619 14:47:01.561028 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:01.560769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37899 (local address 127.0.0.1:54326)
0619 14:47:01.560898 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:01.560954 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260619 14:47:01.561167 7267 consensus_peers.cc:487] T 00000000000000000000000000000000 P 768cb9d7dfb349e3b12896639c2ddd4f -> Peer 4a2a5f1050da4c09b1351fb6ecb3986a (127.0.98.62:37899): Couldn't send request to peer 4a2a5f1050da4c09b1351fb6ecb3986a. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37899: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260619 14:47:01.562310 7804 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e87b561972740938a4f82c45931705b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 768cb9d7dfb349e3b12896639c2ddd4f. Latest consensus state: current_term: 2 leader_uuid: "768cb9d7dfb349e3b12896639c2ddd4f" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4a2a5f1050da4c09b1351fb6ecb3986a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37899 } } peers { permanent_uuid: "8e87b561972740938a4f82c45931705b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36857 } attrs { promote: false } } peers { permanent_uuid: "768cb9d7dfb349e3b12896639c2ddd4f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39275 } attrs { promote: false } } }
I20260619 14:47:01.562487 7804 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e87b561972740938a4f82c45931705b [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:47:01.564486 7813 sys_catalog.cc:455] T 00000000000000000000000000000000 P 768cb9d7dfb349e3b12896639c2ddd4f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "768cb9d7dfb349e3b12896639c2ddd4f" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4a2a5f1050da4c09b1351fb6ecb3986a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37899 } } peers { permanent_uuid: "8e87b561972740938a4f82c45931705b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36857 } attrs { promote: false } } peers { permanent_uuid: "768cb9d7dfb349e3b12896639c2ddd4f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39275 } attrs { promote: false } } }
I20260619 14:47:01.564635 7813 sys_catalog.cc:458] T 00000000000000000000000000000000 P 768cb9d7dfb349e3b12896639c2ddd4f [sys.catalog]: This master's current role is: LEADER
I20260619 14:47:01.564561 7804 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8e87b561972740938a4f82c45931705b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "768cb9d7dfb349e3b12896639c2ddd4f" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4a2a5f1050da4c09b1351fb6ecb3986a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37899 } } peers { permanent_uuid: "8e87b561972740938a4f82c45931705b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36857 } attrs { promote: false } } peers { permanent_uuid: "768cb9d7dfb349e3b12896639c2ddd4f" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39275 } attrs { promote: false } } }
I20260619 14:47:01.564698 7804 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8e87b561972740938a4f82c45931705b [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:47:01.577029 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:01.573122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32954 (local address 127.0.98.3:45697)
0619 14:47:01.573485 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:01.573496 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:01.573510 (+ 14us) server_negotiation.cc:407] Connection header received
0619 14:47:01.573587 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:01.573590 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:01.573642 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:01.573991 (+ 349us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:47:01.574079 (+ 88us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:01.574799 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:01.575978 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:01.576382 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:01.576530 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:01.576599 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:01.576824 (+ 225us) server_negotiation.cc:299] Negotiation successful
0619 14:47:01.576897 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":74,"threads_started":1}
I20260619 14:47:01.577028 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:01.573006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45697 (local address 127.0.0.1:32954)
0619 14:47:01.573151 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:01.573167 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:01.573249 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:01.573776 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:01.573779 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:01.573796 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:47:01.573895 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:01.573903 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:01.574918 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:01.574921 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:01.575845 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:01.575854 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:01.576530 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:01.576533 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:01.576625 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:01.576713 (+ 88us) client_negotiation.cc:769] Sending connection context
0619 14:47:01.576808 (+ 95us) client_negotiation.cc:240] Negotiation successful
0619 14:47:01.576885 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82}
I20260619 14:47:01.577435 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:01.572715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37385 (local address 127.0.0.1:54184)
0619 14:47:01.573071 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:01.573088 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:01.573173 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:01.573667 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:01.573670 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:01.573687 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:47:01.573792 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:01.573798 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:01.574980 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:01.574984 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:01.576094 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:01.576103 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:01.577051 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:01.577058 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:01.577147 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:01.577181 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 14:47:01.577237 (+ 56us) client_negotiation.cc:240] Negotiation successful
0619 14:47:01.577311 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"mutex_wait_us":16}
I20260619 14:47:01.577711 7667 tablet_service.cc:1467] Processing CreateTablet for tablet adce9cadfea94e1481b9aa650a07d14d (DEFAULT_TABLE table=test-table [id=3c71dadc0f9c4d89ad26c6159cf1c6b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:47:01.578048 7667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adce9cadfea94e1481b9aa650a07d14d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:47:01.578047 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:01.572813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54184 (local address 127.0.98.2:37385)
0619 14:47:01.573397 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:01.573402 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:01.573416 (+ 14us) server_negotiation.cc:407] Connection header received
0619 14:47:01.573472 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:01.573476 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:01.573535 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:01.573905 (+ 370us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:47:01.574034 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:01.574852 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:01.576236 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:01.576903 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:01.577735 (+ 832us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:01.577799 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:01.577834 (+ 35us) server_negotiation.cc:299] Negotiation successful
0619 14:47:01.577909 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":507,"thread_start_us":108,"threads_started":1}
I20260619 14:47:01.578533 7531 tablet_service.cc:1467] Processing CreateTablet for tablet adce9cadfea94e1481b9aa650a07d14d (DEFAULT_TABLE table=test-table [id=3c71dadc0f9c4d89ad26c6159cf1c6b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:47:01.578801 7531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adce9cadfea94e1481b9aa650a07d14d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:47:01.578799 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:01.573397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37475 (local address 127.0.0.1:60626)
0619 14:47:01.573676 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:01.573692 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:01.573772 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:01.574217 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:01.574220 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:01.574237 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:47:01.574311 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:01.574316 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:01.575369 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:01.575372 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:01.576980 (+ 1608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:01.576992 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:01.578367 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:01.578370 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:01.578451 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:01.578473 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 14:47:01.578581 (+ 108us) client_negotiation.cc:240] Negotiation successful
0619 14:47:01.578658 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"mutex_wait_us":25,"thread_start_us":122,"threads_started":1}
I20260619 14:47:01.579188 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:01.573729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60626 (local address 127.0.98.1:37475)
0619 14:47:01.573989 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:01.573994 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:01.574008 (+ 14us) server_negotiation.cc:407] Connection header received
0619 14:47:01.574056 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:01.574058 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:01.574101 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:01.574188 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:47:01.574434 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:01.575243 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:01.577475 (+ 2232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:01.578135 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:01.578245 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:01.578307 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:01.578812 (+ 505us) server_negotiation.cc:299] Negotiation successful
0619 14:47:01.579059 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":91,"threads_started":1}
I20260619 14:47:01.579906 7394 tablet_service.cc:1467] Processing CreateTablet for tablet adce9cadfea94e1481b9aa650a07d14d (DEFAULT_TABLE table=test-table [id=3c71dadc0f9c4d89ad26c6159cf1c6b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:47:01.580382 7394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adce9cadfea94e1481b9aa650a07d14d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:47:01.581422 7825 tablet_bootstrap.cc:492] T adce9cadfea94e1481b9aa650a07d14d P 0f513b290a8c49ee81a89fa8d2152c04: Bootstrap starting.
I20260619 14:47:01.582023 7826 tablet_bootstrap.cc:492] T adce9cadfea94e1481b9aa650a07d14d P 2cb8830304e145db8591b8ff765eeef3: Bootstrap starting.
I20260619 14:47:01.582361 7825 tablet_bootstrap.cc:654] T adce9cadfea94e1481b9aa650a07d14d P 0f513b290a8c49ee81a89fa8d2152c04: Neither blocks nor log segments found. Creating new log.
I20260619 14:47:01.582996 7826 tablet_bootstrap.cc:654] T adce9cadfea94e1481b9aa650a07d14d P 2cb8830304e145db8591b8ff765eeef3: Neither blocks nor log segments found. Creating new log.
I20260619 14:47:01.583585 7828 tablet_bootstrap.cc:492] T adce9cadfea94e1481b9aa650a07d14d P 1b116e7a13764f34b66cdbb5e53338e4: Bootstrap starting.
I20260619 14:47:01.584309 7825 tablet_bootstrap.cc:492] T adce9cadfea94e1481b9aa650a07d14d P 0f513b290a8c49ee81a89fa8d2152c04: No bootstrap required, opened a new log
I20260619 14:47:01.584410 7825 ts_tablet_manager.cc:1397] T adce9cadfea94e1481b9aa650a07d14d P 0f513b290a8c49ee81a89fa8d2152c04: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260619 14:47:01.584563 7826 tablet_bootstrap.cc:492] T adce9cadfea94e1481b9aa650a07d14d P 2cb8830304e145db8591b8ff765eeef3: No bootstrap required, opened a new log
I20260619 14:47:01.584630 7828 tablet_bootstrap.cc:654] T adce9cadfea94e1481b9aa650a07d14d P 1b116e7a13764f34b66cdbb5e53338e4: Neither blocks nor log segments found. Creating new log.
I20260619 14:47:01.584730 7826 ts_tablet_manager.cc:1397] T adce9cadfea94e1481b9aa650a07d14d P 2cb8830304e145db8591b8ff765eeef3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 14:47:01.584965 7825 raft_consensus.cc:348] T adce9cadfea94e1481b9aa650a07d14d P 0f513b290a8c49ee81a89fa8d2152c04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f513b290a8c49ee81a89fa8d2152c04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37385 } } peers { permanent_uuid: "2cb8830304e145db8591b8ff765eeef3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45697 } } peers { permanent_uuid: "1b116e7a13764f34b66cdbb5e53338e4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37475 } }
I20260619 14:47:01.585112 7825 raft_consensus.cc:374] T adce9cadfea94e1481b9aa650a07d14d P 0f513b290a8c49ee81a89fa8d2152c04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:47:01.585150 7825 raft_consensus.cc:729] T adce9cadfea94e1481b9aa650a07d14d P 0f513b290a8c49ee81a89fa8d2152c04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f513b290a8c49ee81a89fa8d2152c04, State: Initialized, Role: FOLLOWER
I20260619 14:47:01.585179 7826 raft_consensus.cc:348] T adce9cadfea94e1481b9aa650a07d14d P 2cb8830304e145db8591b8ff765eeef3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f513b290a8c49ee81a89fa8d2152c04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37385 } } peers { permanent_uuid: "2cb8830304e145db8591b8ff765eeef3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45697 } } peers { permanent_uuid: "1b116e7a13764f34b66cdbb5e53338e4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37475 } }
I20260619 14:47:01.585312 7826 raft_consensus.cc:374] T adce9cadfea94e1481b9aa650a07d14d P 2cb8830304e145db8591b8ff765eeef3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:47:01.585367 7826 raft_consensus.cc:729] T adce9cadfea94e1481b9aa650a07d14d P 2cb8830304e145db8591b8ff765eeef3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cb8830304e145db8591b8ff765eeef3, State: Initialized, Role: FOLLOWER
I20260619 14:47:01.585328 7825 consensus_queue.cc:260] T adce9cadfea94e1481b9aa650a07d14d P 0f513b290a8c49ee81a89fa8d2152c04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f513b290a8c49ee81a89fa8d2152c04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37385 } } peers { permanent_uuid: "2cb8830304e145db8591b8ff765eeef3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45697 } } peers { permanent_uuid: "1b116e7a13764f34b66cdbb5e53338e4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37475 } }
I20260619 14:47:01.585533 7826 consensus_queue.cc:260] T adce9cadfea94e1481b9aa650a07d14d P 2cb8830304e145db8591b8ff765eeef3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f513b290a8c49ee81a89fa8d2152c04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37385 } } peers { permanent_uuid: "2cb8830304e145db8591b8ff765eeef3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45697 } } peers { permanent_uuid: "1b116e7a13764f34b66cdbb5e53338e4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37475 } }
I20260619 14:47:01.585711 7825 ts_tablet_manager.cc:1428] T adce9cadfea94e1481b9aa650a07d14d P 0f513b290a8c49ee81a89fa8d2152c04: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260619 14:47:01.585907 7828 tablet_bootstrap.cc:492] T adce9cadfea94e1481b9aa650a07d14d P 1b116e7a13764f34b66cdbb5e53338e4: No bootstrap required, opened a new log
I20260619 14:47:01.585919 7826 ts_tablet_manager.cc:1428] T adce9cadfea94e1481b9aa650a07d14d P 2cb8830304e145db8591b8ff765eeef3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:47:01.586004 7828 ts_tablet_manager.cc:1397] T adce9cadfea94e1481b9aa650a07d14d P 1b116e7a13764f34b66cdbb5e53338e4: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260619 14:47:01.586493 7828 raft_consensus.cc:348] T adce9cadfea94e1481b9aa650a07d14d P 1b116e7a13764f34b66cdbb5e53338e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f513b290a8c49ee81a89fa8d2152c04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37385 } } peers { permanent_uuid: "2cb8830304e145db8591b8ff765eeef3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45697 } } peers { permanent_uuid: "1b116e7a13764f34b66cdbb5e53338e4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37475 } }
I20260619 14:47:01.586598 7828 raft_consensus.cc:374] T adce9cadfea94e1481b9aa650a07d14d P 1b116e7a13764f34b66cdbb5e53338e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:47:01.586611 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:01.586149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37899 (local address 127.0.98.2:58149)
0619 14:47:01.586425 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:01.586505 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":117,"threads_started":1}
I20260619 14:47:01.586635 7828 raft_consensus.cc:729] T adce9cadfea94e1481b9aa650a07d14d P 1b116e7a13764f34b66cdbb5e53338e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b116e7a13764f34b66cdbb5e53338e4, State: Initialized, Role: FOLLOWER
I20260619 14:47:01.586767 7828 consensus_queue.cc:260] T adce9cadfea94e1481b9aa650a07d14d P 1b116e7a13764f34b66cdbb5e53338e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f513b290a8c49ee81a89fa8d2152c04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37385 } } peers { permanent_uuid: "2cb8830304e145db8591b8ff765eeef3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45697 } } peers { permanent_uuid: "1b116e7a13764f34b66cdbb5e53338e4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37475 } }
I20260619 14:47:01.587126 7828 ts_tablet_manager.cc:1428] T adce9cadfea94e1481b9aa650a07d14d P 1b116e7a13764f34b66cdbb5e53338e4: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260619 14:47:01.587545 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:01.587011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37899 (local address 127.0.98.2:38955)
0619 14:47:01.587397 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:01.587449 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":82,"threads_started":1}
I20260619 14:47:01.589480 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:01.589051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37899 (local address 127.0.98.1:43005)
0619 14:47:01.589300 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:01.589359 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":104,"threads_started":1}
I20260619 14:47:01.590770 7598 heartbeater.cc:503] Master 127.0.98.60:39275 was elected leader, sending a full tablet report...
I20260619 14:47:01.590816 7461 heartbeater.cc:503] Master 127.0.98.60:39275 was elected leader, sending a full tablet report...
I20260619 14:47:01.591298 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:01.590657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37899 (local address 127.0.98.1:36147)
0619 14:47:01.591169 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:01.591223 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":434}
I20260619 14:47:01.592854 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:01.586735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37899 (local address 127.0.98.3:38083)
0619 14:47:01.592628 (+ 5893us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:01.592725 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5809,"thread_start_us":101,"threads_started":1}
I20260619 14:47:01.592972 7734 heartbeater.cc:503] Master 127.0.98.60:39275 was elected leader, sending a full tablet report...
I20260619 14:47:01.592980 7838 raft_consensus.cc:482] T adce9cadfea94e1481b9aa650a07d14d P 2cb8830304e145db8591b8ff765eeef3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:47:01.593456 7838 raft_consensus.cc:504] T adce9cadfea94e1481b9aa650a07d14d P 2cb8830304e145db8591b8ff765eeef3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f513b290a8c49ee81a89fa8d2152c04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37385 } } peers { permanent_uuid: "2cb8830304e145db8591b8ff765eeef3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45697 } } peers { permanent_uuid: "1b116e7a13764f34b66cdbb5e53338e4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37475 } }
I20260619 14:47:01.594077 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:01.593815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37899 (local address 127.0.98.3:42293)
0619 14:47:01.593946 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:01.594005 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260619 14:47:01.595000 7838 leader_election.cc:288] T adce9cadfea94e1481b9aa650a07d14d P 2cb8830304e145db8591b8ff765eeef3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f513b290a8c49ee81a89fa8d2152c04 (127.0.98.2:37385), 1b116e7a13764f34b66cdbb5e53338e4 (127.0.98.1:37475)
I20260619 14:47:01.598773 7831 raft_consensus.cc:482] T adce9cadfea94e1481b9aa650a07d14d P 0f513b290a8c49ee81a89fa8d2152c04 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:47:01.598989 7831 raft_consensus.cc:504] T adce9cadfea94e1481b9aa650a07d14d P 0f513b290a8c49ee81a89fa8d2152c04 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f513b290a8c49ee81a89fa8d2152c04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37385 } } peers { permanent_uuid: "2cb8830304e145db8591b8ff765eeef3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45697 } } peers { permanent_uuid: "1b116e7a13764f34b66cdbb5e53338e4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37475 } }
I20260619 14:47:01.599802 7687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adce9cadfea94e1481b9aa650a07d14d" candidate_uuid: "0f513b290a8c49ee81a89fa8d2152c04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cb8830304e145db8591b8ff765eeef3" is_pre_election: true
I20260619 14:47:01.599989 7687 raft_consensus.cc:2454] T adce9cadfea94e1481b9aa650a07d14d P 2cb8830304e145db8591b8ff765eeef3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f513b290a8c49ee81a89fa8d2152c04 in term 0.
I20260619 14:47:01.600122 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:01.595340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47993 (local address 127.0.98.2:37385)
0619 14:47:01.595456 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:01.595462 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:01.595597 (+ 135us) server_negotiation.cc:407] Connection header received
0619 14:47:01.595733 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:01.595735 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:01.595778 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:01.595882 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:47:01.596178 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:01.597057 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:01.598516 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:01.599268 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:01.599398 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:01.599468 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:01.599887 (+ 419us) server_negotiation.cc:299] Negotiation successful
0619 14:47:01.599983 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 14:47:01.600134 7414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adce9cadfea94e1481b9aa650a07d14d" candidate_uuid: "0f513b290a8c49ee81a89fa8d2152c04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b116e7a13764f34b66cdbb5e53338e4" is_pre_election: true
I20260619 14:47:01.600260 7414 raft_consensus.cc:2454] T adce9cadfea94e1481b9aa650a07d14d P 1b116e7a13764f34b66cdbb5e53338e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f513b290a8c49ee81a89fa8d2152c04 in term 0.
I20260619 14:47:01.600332 7484 leader_election.cc:302] T adce9cadfea94e1481b9aa650a07d14d P 0f513b290a8c49ee81a89fa8d2152c04 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f513b290a8c49ee81a89fa8d2152c04, 2cb8830304e145db8591b8ff765eeef3; no voters:
I20260619 14:47:01.600571 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:01.595265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37385 (local address 127.0.98.3:47993)
0619 14:47:01.595510 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:01.595525 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:01.595640 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:01.595895 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:01.595899 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:01.595917 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:47:01.596028 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:01.596035 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:01.597208 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:01.597211 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:01.598368 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:01.598377 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:01.599411 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:01.599415 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:01.599717 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:01.599744 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 14:47:01.600309 (+ 565us) client_negotiation.cc:240] Negotiation successful
0619 14:47:01.600406 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155}
I20260619 14:47:01.600904 7840 raft_consensus.cc:2781] T adce9cadfea94e1481b9aa650a07d14d P 0f513b290a8c49ee81a89fa8d2152c04 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:47:01.600997 7840 raft_consensus.cc:482] T adce9cadfea94e1481b9aa650a07d14d P 0f513b290a8c49ee81a89fa8d2152c04 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:47:01.601152 7551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adce9cadfea94e1481b9aa650a07d14d" candidate_uuid: "2cb8830304e145db8591b8ff765eeef3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f513b290a8c49ee81a89fa8d2152c04" is_pre_election: true
I20260619 14:47:01.601359 7840 raft_consensus.cc:3037] T adce9cadfea94e1481b9aa650a07d14d P 0f513b290a8c49ee81a89fa8d2152c04 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:47:01.601619 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:01.596032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42623 (local address 127.0.98.1:37475)
0619 14:47:01.596195 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:01.596200 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:01.596216 (+ 16us) server_negotiation.cc:407] Connection header received
0619 14:47:01.596264 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:01.596267 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:01.596315 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:01.596394 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:47:01.596918 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:01.597724 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:01.599242 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:01.600027 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:01.600552 (+ 525us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:01.600815 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:01.601362 (+ 547us) server_negotiation.cc:299] Negotiation successful
0619 14:47:01.601484 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 14:47:01.601723 7831 leader_election.cc:288] T adce9cadfea94e1481b9aa650a07d14d P 0f513b290a8c49ee81a89fa8d2152c04 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2cb8830304e145db8591b8ff765eeef3 (127.0.98.3:45697), 1b116e7a13764f34b66cdbb5e53338e4 (127.0.98.1:37475)
I20260619 14:47:01.602044 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:01.595641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37475 (local address 127.0.98.3:42623)
0619 14:47:01.595901 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:01.595916 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:01.596017 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:01.596629 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:01.596633 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:01.596679 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:47:01.596784 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:01.596794 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:01.597889 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:01.597892 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:01.599035 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:01.599046 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:01.600876 (+ 1830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:01.600879 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:01.600970 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:01.601182 (+ 212us) client_negotiation.cc:769] Sending connection context
0619 14:47:01.601260 (+ 78us) client_negotiation.cc:240] Negotiation successful
0619 14:47:01.601931 (+ 671us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":133,"threads_started":1}
I20260619 14:47:01.602582 7414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adce9cadfea94e1481b9aa650a07d14d" candidate_uuid: "2cb8830304e145db8591b8ff765eeef3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b116e7a13764f34b66cdbb5e53338e4" is_pre_election: true
I20260619 14:47:01.602753 7414 raft_consensus.cc:2454] T adce9cadfea94e1481b9aa650a07d14d P 1b116e7a13764f34b66cdbb5e53338e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2cb8830304e145db8591b8ff765eeef3 in term 0.
I20260619 14:47:01.602993 7840 raft_consensus.cc:504] T adce9cadfea94e1481b9aa650a07d14d P 0f513b290a8c49ee81a89fa8d2152c04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f513b290a8c49ee81a89fa8d2152c04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37385 } } peers { permanent_uuid: "2cb8830304e145db8591b8ff765eeef3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45697 } } peers { permanent_uuid: "1b116e7a13764f34b66cdbb5e53338e4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37475 } }
I20260619 14:47:01.603190 7623 leader_election.cc:302] T adce9cadfea94e1481b9aa650a07d14d P 2cb8830304e145db8591b8ff765eeef3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b116e7a13764f34b66cdbb5e53338e4, 2cb8830304e145db8591b8ff765eeef3; no voters:
I20260619 14:47:01.603294 7551 raft_consensus.cc:2379] T adce9cadfea94e1481b9aa650a07d14d P 0f513b290a8c49ee81a89fa8d2152c04 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2cb8830304e145db8591b8ff765eeef3 in current term 1: Already voted for candidate 0f513b290a8c49ee81a89fa8d2152c04 in this term.
I20260619 14:47:01.603588 7840 leader_election.cc:288] T adce9cadfea94e1481b9aa650a07d14d P 0f513b290a8c49ee81a89fa8d2152c04 [CANDIDATE]: Term 1 election: Requested vote from peers 2cb8830304e145db8591b8ff765eeef3 (127.0.98.3:45697), 1b116e7a13764f34b66cdbb5e53338e4 (127.0.98.1:37475)
I20260619 14:47:01.603690 7838 raft_consensus.cc:2781] T adce9cadfea94e1481b9aa650a07d14d P 2cb8830304e145db8591b8ff765eeef3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:47:01.603761 7838 raft_consensus.cc:482] T adce9cadfea94e1481b9aa650a07d14d P 2cb8830304e145db8591b8ff765eeef3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:47:01.603812 7838 raft_consensus.cc:3037] T adce9cadfea94e1481b9aa650a07d14d P 2cb8830304e145db8591b8ff765eeef3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:47:01.603847 7687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adce9cadfea94e1481b9aa650a07d14d" candidate_uuid: "0f513b290a8c49ee81a89fa8d2152c04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cb8830304e145db8591b8ff765eeef3"
I20260619 14:47:01.604383 7414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adce9cadfea94e1481b9aa650a07d14d" candidate_uuid: "0f513b290a8c49ee81a89fa8d2152c04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b116e7a13764f34b66cdbb5e53338e4"
I20260619 14:47:01.604498 7414 raft_consensus.cc:3037] T adce9cadfea94e1481b9aa650a07d14d P 1b116e7a13764f34b66cdbb5e53338e4 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:47:01.605165 7838 raft_consensus.cc:504] T adce9cadfea94e1481b9aa650a07d14d P 2cb8830304e145db8591b8ff765eeef3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f513b290a8c49ee81a89fa8d2152c04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37385 } } peers { permanent_uuid: "2cb8830304e145db8591b8ff765eeef3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45697 } } peers { permanent_uuid: "1b116e7a13764f34b66cdbb5e53338e4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37475 } }
I20260619 14:47:01.605474 7687 raft_consensus.cc:2379] T adce9cadfea94e1481b9aa650a07d14d P 2cb8830304e145db8591b8ff765eeef3 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 0f513b290a8c49ee81a89fa8d2152c04 in current term 1: Already voted for candidate 2cb8830304e145db8591b8ff765eeef3 in this term.
I20260619 14:47:01.605620 7838 leader_election.cc:288] T adce9cadfea94e1481b9aa650a07d14d P 2cb8830304e145db8591b8ff765eeef3 [CANDIDATE]: Term 1 election: Requested vote from peers 0f513b290a8c49ee81a89fa8d2152c04 (127.0.98.2:37385), 1b116e7a13764f34b66cdbb5e53338e4 (127.0.98.1:37475)
I20260619 14:47:01.605700 7414 raft_consensus.cc:2454] T adce9cadfea94e1481b9aa650a07d14d P 1b116e7a13764f34b66cdbb5e53338e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f513b290a8c49ee81a89fa8d2152c04 in term 1.
I20260619 14:47:01.606098 7487 leader_election.cc:302] T adce9cadfea94e1481b9aa650a07d14d P 0f513b290a8c49ee81a89fa8d2152c04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f513b290a8c49ee81a89fa8d2152c04, 1b116e7a13764f34b66cdbb5e53338e4; no voters:
I20260619 14:47:01.606117 7551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adce9cadfea94e1481b9aa650a07d14d" candidate_uuid: "2cb8830304e145db8591b8ff765eeef3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f513b290a8c49ee81a89fa8d2152c04"
I20260619 14:47:01.606231 7414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adce9cadfea94e1481b9aa650a07d14d" candidate_uuid: "2cb8830304e145db8591b8ff765eeef3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b116e7a13764f34b66cdbb5e53338e4"
I20260619 14:47:01.606292 7551 raft_consensus.cc:2379] T adce9cadfea94e1481b9aa650a07d14d P 0f513b290a8c49ee81a89fa8d2152c04 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 2cb8830304e145db8591b8ff765eeef3 in current term 1: Already voted for candidate 0f513b290a8c49ee81a89fa8d2152c04 in this term.
I20260619 14:47:01.606366 7840 raft_consensus.cc:2781] T adce9cadfea94e1481b9aa650a07d14d P 0f513b290a8c49ee81a89fa8d2152c04 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:47:01.606396 7414 raft_consensus.cc:2379] T adce9cadfea94e1481b9aa650a07d14d P 1b116e7a13764f34b66cdbb5e53338e4 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 2cb8830304e145db8591b8ff765eeef3 in current term 1: Already voted for candidate 0f513b290a8c49ee81a89fa8d2152c04 in this term.
I20260619 14:47:01.606479 7840 raft_consensus.cc:686] T adce9cadfea94e1481b9aa650a07d14d P 0f513b290a8c49ee81a89fa8d2152c04 [term 1 LEADER]: Becoming Leader. State: Replica: 0f513b290a8c49ee81a89fa8d2152c04, State: Running, Role: LEADER
I20260619 14:47:01.606631 7840 consensus_queue.cc:237] T adce9cadfea94e1481b9aa650a07d14d P 0f513b290a8c49ee81a89fa8d2152c04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f513b290a8c49ee81a89fa8d2152c04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37385 } } peers { permanent_uuid: "2cb8830304e145db8591b8ff765eeef3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45697 } } peers { permanent_uuid: "1b116e7a13764f34b66cdbb5e53338e4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37475 } }
I20260619 14:47:01.606726 7623 leader_election.cc:302] T adce9cadfea94e1481b9aa650a07d14d P 2cb8830304e145db8591b8ff765eeef3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2cb8830304e145db8591b8ff765eeef3; no voters: 0f513b290a8c49ee81a89fa8d2152c04, 1b116e7a13764f34b66cdbb5e53338e4
I20260619 14:47:01.606922 7838 raft_consensus.cc:2726] T adce9cadfea94e1481b9aa650a07d14d P 2cb8830304e145db8591b8ff765eeef3 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260619 14:47:01.607220 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:01.606904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37899 (local address 127.0.98.2:41135)
0619 14:47:01.607060 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:01.607128 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260619 14:47:01.609077 7278 catalog_manager.cc:5495] T adce9cadfea94e1481b9aa650a07d14d P 0f513b290a8c49ee81a89fa8d2152c04 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0f513b290a8c49ee81a89fa8d2152c04 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "0f513b290a8c49ee81a89fa8d2152c04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f513b290a8c49ee81a89fa8d2152c04" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37385 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2cb8830304e145db8591b8ff765eeef3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b116e7a13764f34b66cdbb5e53338e4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37475 } health_report { overall_health: UNKNOWN } } }
I20260619 14:47:01.626271 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:01.622746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54192 (local address 127.0.98.2:37385)
0619 14:47:01.622897 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:01.622902 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:01.622918 (+ 16us) server_negotiation.cc:407] Connection header received
0619 14:47:01.623091 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:01.623093 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:01.623138 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:01.623223 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:47:01.623513 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:01.624042 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:01.624966 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:01.625209 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:01.625328 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:01.625757 (+ 429us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:47:01.625834 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:47:01.625920 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:01.626043 (+ 123us) server_negotiation.cc:299] Negotiation successful
0619 14:47:01.626102 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 14:47:01.628885 7414 raft_consensus.cc:1261] T adce9cadfea94e1481b9aa650a07d14d P 1b116e7a13764f34b66cdbb5e53338e4 [term 1 FOLLOWER]: Refusing update from remote peer 0f513b290a8c49ee81a89fa8d2152c04: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:47:01.628909 7687 raft_consensus.cc:1261] T adce9cadfea94e1481b9aa650a07d14d P 2cb8830304e145db8591b8ff765eeef3 [term 1 FOLLOWER]: Refusing update from remote peer 0f513b290a8c49ee81a89fa8d2152c04: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:47:01.629591 7831 consensus_queue.cc:1035] T adce9cadfea94e1481b9aa650a07d14d P 0f513b290a8c49ee81a89fa8d2152c04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cb8830304e145db8591b8ff765eeef3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:47:01.629817 7840 consensus_queue.cc:1035] T adce9cadfea94e1481b9aa650a07d14d P 0f513b290a8c49ee81a89fa8d2152c04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b116e7a13764f34b66cdbb5e53338e4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:47:01.632344 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:01.631793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37899 (local address 127.0.98.3:40473)
0619 14:47:01.632237 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:01.632282 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372}
I20260619 14:47:01.632344 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:01.631240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37899 (local address 127.0.98.1:41357)
0619 14:47:01.632188 (+ 948us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:01.632254 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":867}
I20260619 14:47:01.634146 7844 mvcc.cc:205] Tried to move back new op lower bound from 7298582206987083776 to 7298582206902554624. Current Snapshot: MvccSnapshot[applied={T|T < 7298582206987083776}]
I20260619 14:47:01.634858 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:01.634413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37899 (local address 127.0.98.2:47199)
0619 14:47:01.634711 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:01.634771 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226}
I20260619 14:47:01.635533 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:01.635058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37899 (local address 127.0.98.1:39841)
0619 14:47:01.635394 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:01.635459 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":265}
I20260619 14:47:01.636303 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:01.635986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37899 (local address 127.0.98.2:55555)
0619 14:47:01.636154 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:01.636232 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260619 14:47:01.638661 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:01.638210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37899 (local address 127.0.98.3:55165)
0619 14:47:01.638535 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:01.638592 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240}
I20260619 14:47:01.642431 7278 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34586:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:47:01.642614 7278 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34586:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:47:01.647536 7278 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 768cb9d7dfb349e3b12896639c2ddd4f: Sending DeleteTablet for 3 replicas of tablet adce9cadfea94e1481b9aa650a07d14d
I20260619 14:47:01.648525 7667 tablet_service.cc:1514] Processing DeleteTablet for tablet adce9cadfea94e1481b9aa650a07d14d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:47:01 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:32954
I20260619 14:47:01.648795 7394 tablet_service.cc:1514] Processing DeleteTablet for tablet adce9cadfea94e1481b9aa650a07d14d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:47:01 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:60626
I20260619 14:47:01.648870 7531 tablet_service.cc:1514] Processing DeleteTablet for tablet adce9cadfea94e1481b9aa650a07d14d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:47:01 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:54184
I20260619 14:47:01.649255 7854 tablet_replica.cc:330] T adce9cadfea94e1481b9aa650a07d14d P 1b116e7a13764f34b66cdbb5e53338e4: stopping tablet replica
I20260619 14:47:01.649339 7855 tablet_replica.cc:330] T adce9cadfea94e1481b9aa650a07d14d P 0f513b290a8c49ee81a89fa8d2152c04: stopping tablet replica
I20260619 14:47:01.649396 7856 tablet_replica.cc:330] T adce9cadfea94e1481b9aa650a07d14d P 2cb8830304e145db8591b8ff765eeef3: stopping tablet replica
I20260619 14:47:01.649466 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 7242
I20260619 14:47:01.649479 7855 raft_consensus.cc:2229] T adce9cadfea94e1481b9aa650a07d14d P 0f513b290a8c49ee81a89fa8d2152c04 [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:47:01.649556 7856 raft_consensus.cc:2229] T adce9cadfea94e1481b9aa650a07d14d P 2cb8830304e145db8591b8ff765eeef3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:47:01.649591 7854 raft_consensus.cc:2229] T adce9cadfea94e1481b9aa650a07d14d P 1b116e7a13764f34b66cdbb5e53338e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:47:01.649716 7855 raft_consensus.cc:2258] T adce9cadfea94e1481b9aa650a07d14d P 0f513b290a8c49ee81a89fa8d2152c04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:47:01.651281 7856 raft_consensus.cc:2258] T adce9cadfea94e1481b9aa650a07d14d P 2cb8830304e145db8591b8ff765eeef3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:47:01.651417 7855 ts_tablet_manager.cc:1905] T adce9cadfea94e1481b9aa650a07d14d P 0f513b290a8c49ee81a89fa8d2152c04: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:47:01.652210 7856 ts_tablet_manager.cc:1905] T adce9cadfea94e1481b9aa650a07d14d P 2cb8830304e145db8591b8ff765eeef3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:47:01.654662 7856 ts_tablet_manager.cc:1918] T adce9cadfea94e1481b9aa650a07d14d P 2cb8830304e145db8591b8ff765eeef3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:47:01.654786 7856 log.cc:1196] T adce9cadfea94e1481b9aa650a07d14d P 2cb8830304e145db8591b8ff765eeef3: Deleting WAL directory at /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/ts-2/wal/wals/adce9cadfea94e1481b9aa650a07d14d
I20260619 14:47:01.655112 7856 ts_tablet_manager.cc:1939] T adce9cadfea94e1481b9aa650a07d14d P 2cb8830304e145db8591b8ff765eeef3: Deleting consensus metadata
I20260619 14:47:01.655182 7855 ts_tablet_manager.cc:1918] T adce9cadfea94e1481b9aa650a07d14d P 0f513b290a8c49ee81a89fa8d2152c04: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:47:01.655274 7855 log.cc:1196] T adce9cadfea94e1481b9aa650a07d14d P 0f513b290a8c49ee81a89fa8d2152c04: Deleting WAL directory at /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781880383132139-392-0/minicluster-data/ts-1/wal/wals/adce9cadfea94e1481b9aa650a07d14d
I20260619 14:47:01.655591 7855 ts_tablet_manager.cc:1939] T adce9cadfea94e1481b9aa650a07d14d P 0f513b290a8c49ee81a89fa8d2152c04: Deleting consensus metadata
I20260619 14:47:01.656028 7265 catalog_manager.cc:4841] TS 2cb8830304e145db8591b8ff765eeef3 (127.0.98.3:45697): tablet adce9cadfea94e1481b9aa650a07d14d (table test-table [id=3c71dadc0f9c4d89ad26c6159cf1c6b7]) successfully deleted
I20260619 14:47:01.656426 7266 catalog_manager.cc:4841] TS 0f513b290a8c49ee81a89fa8d2152c04 (127.0.98.2:37385): tablet adce9cadfea94e1481b9aa650a07d14d (table test-table [id=3c71dadc0f9c4d89ad26c6159cf1c6b7]) successfully deleted
I20260619 14:47:01.658383 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 7468
I20260619 14:47:01.659241 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:01.658821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37475 (local address 127.0.0.1:60642)
0619 14:47:01.659075 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:01.659132 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155}
W20260619 14:47:01.659569 7268 catalog_manager.cc:4584] TS 1b116e7a13764f34b66cdbb5e53338e4 (127.0.98.1:37475): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet adce9cadfea94e1481b9aa650a07d14d: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37475: connect: Connection refused (error 111)
I20260619 14:47:01.666893 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 7604
I20260619 14:47:01.673499 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 6961
I20260619 14:47:01.680117 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 7069
2026-06-19T14:47:01Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5748 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880383132139-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:47:01 dist-test-slave-2sb1 krb5kdc[7859](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:47:01 dist-test-slave-2sb1 krb5kdc[7859](info): set up 2 sockets
Jun 19 14:47:01 dist-test-slave-2sb1 krb5kdc[7859](info): commencing operation
krb5kdc: starting...
W20260619 14:47:02.802011 392 mini_kdc.cc:121] Time spent starting KDC: real 1.097s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880383132139-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880383132139-392-0/krb5kdc/test-user.keytab.
Jun 19 14:47:02 dist-test-slave-2sb1 krb5kdc[7859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880422, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:47:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:47:02Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880383132139-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880383132139-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260619 14:47:02.953768 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880383132139-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880383132139-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880383132139-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39425
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880383132139-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39425
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:47:03.082648 7872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:47:03.083007 7872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:47:03.083099 7872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:47:03.087502 7872 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:47:03.087620 7872 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260619 14:47:03.087651 7872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:47:03.087695 7872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:47:03.087736 7872 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:47:03.087793 7872 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:47:03.093910 7872 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880383132139-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880383132139-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39425
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880383132139-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39425
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7872
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880383132139-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:47:03.095144 7872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:47:03.096247 7872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:47:03.105599 7878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:47:03.105587 7880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:47:03.105568 7877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:47:03.106300 7872 server_base.cc:1034] running on GCE node
I20260619 14:47:03.106917 7872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:47:03.108141 7872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:47:03.109349 7872 hybrid_clock.cc:648] HybridClock initialized: now 1781880423109313 us; error 52 us; skew 500 ppm
Jun 19 14:47:03 dist-test-slave-2sb1 krb5kdc[7859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880423, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:47:03.112784 7872 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260619 14:47:03.113948 7872 webserver.cc:460] Webserver started at http://127.0.98.62:39751/ using document root <none> and password file <none>
I20260619 14:47:03.114622 7872 fs_manager.cc:359] Metadata directory not provided
I20260619 14:47:03.114727 7872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:47:03.114990 7872 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:47:03.117710 7872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880383132139-392-0/minicluster-data/master-0/data/instance:
uuid: "e3f4029a0c9b421faf47effdceacd7b5"
format_stamp: "Formatted at 2026-06-19 14:47:03 on dist-test-slave-2sb1"
server_key: "b2ff091102a5705e9498230f06d5ed5b"
server_key_iv: "7a8120215ac616dc974ffa7ad494d56b"
server_key_version: "encryptionkey@0"
I20260619 14:47:03.118463 7872 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880383132139-392-0/minicluster-data/master-0/wal/instance:
uuid: "e3f4029a0c9b421faf47effdceacd7b5"
format_stamp: "Formatted at 2026-06-19 14:47:03 on dist-test-slave-2sb1"
server_key: "b2ff091102a5705e9498230f06d5ed5b"
server_key_iv: "7a8120215ac616dc974ffa7ad494d56b"
server_key_version: "encryptionkey@0"
I20260619 14:47:03.123027 7872 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 14:47:03.125927 7887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:47:03.127506 7872 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:47:03.127722 7872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880383132139-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880383132139-392-0/minicluster-data/master-0/wal
uuid: "e3f4029a0c9b421faf47effdceacd7b5"
format_stamp: "Formatted at 2026-06-19 14:47:03 on dist-test-slave-2sb1"
server_key: "b2ff091102a5705e9498230f06d5ed5b"
server_key_iv: "7a8120215ac616dc974ffa7ad494d56b"
server_key_version: "encryptionkey@0"
I20260619 14:47:03.127864 7872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880383132139-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880383132139-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880383132139-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:47:03.144068 7872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:47:03.148366 7872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:47:03.148639 7872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:47:03.158790 7872 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39425
I20260619 14:47:03.158824 7949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39425 every 8 connection(s)
I20260619 14:47:03.159828 7872 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:47:03.162017 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 7872
I20260619 14:47:03.162168 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880383132139-392-0/minicluster-data/master-0/wal/instance
I20260619 14:47:03.162506 392 external_mini_cluster.cc:1428] Setting key 98d5233b288f5a74beb209252cffc771
I20260619 14:47:03.163374 7950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:47:03.171656 7950 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3f4029a0c9b421faf47effdceacd7b5: Bootstrap starting.
Jun 19 14:47:03 dist-test-slave-2sb1 krb5kdc[7859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880422, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 14:47:03.174611 7950 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e3f4029a0c9b421faf47effdceacd7b5: Neither blocks nor log segments found. Creating new log.
I20260619 14:47:03.176127 7950 log.cc:824] T 00000000000000000000000000000000 P e3f4029a0c9b421faf47effdceacd7b5: Log is configured to *not* fsync() on all Append() calls
I20260619 14:47:03.179185 7950 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3f4029a0c9b421faf47effdceacd7b5: No bootstrap required, opened a new log
I20260619 14:47:03.180593 7953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:03.164406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48920 (local address 127.0.98.62:39425)
0619 14:47:03.164971 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:03.164987 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:03.165177 (+ 190us) server_negotiation.cc:407] Connection header received
0619 14:47:03.165996 (+ 819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:03.166080 (+ 84us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:03.166440 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:03.166788 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:03.167834 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:03.169011 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:03.170359 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:03.170927 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:03.171062 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:03.174416 (+ 3354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:47:03.174425 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:47:03.174458 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:47:03.174513 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:47:03.176896 (+ 2383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:03.177509 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:03.177516 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:03.177523 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:03.177688 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:03.178010 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:03.178013 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:03.178015 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:03.178696 (+ 681us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:47:03.178944 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:03.179203 (+ 259us) server_negotiation.cc:299] Negotiation successful
0619 14:47:03.179542 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":190,"threads_started":1}
I20260619 14:47:03.183354 7950 raft_consensus.cc:348] T 00000000000000000000000000000000 P e3f4029a0c9b421faf47effdceacd7b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3f4029a0c9b421faf47effdceacd7b5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39425 } }
I20260619 14:47:03.183617 7950 raft_consensus.cc:374] T 00000000000000000000000000000000 P e3f4029a0c9b421faf47effdceacd7b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:47:03.183657 7950 raft_consensus.cc:729] T 00000000000000000000000000000000 P e3f4029a0c9b421faf47effdceacd7b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3f4029a0c9b421faf47effdceacd7b5, State: Initialized, Role: FOLLOWER
I20260619 14:47:03.184119 7950 consensus_queue.cc:260] T 00000000000000000000000000000000 P e3f4029a0c9b421faf47effdceacd7b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3f4029a0c9b421faf47effdceacd7b5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39425 } }
I20260619 14:47:03.184257 7950 raft_consensus.cc:388] T 00000000000000000000000000000000 P e3f4029a0c9b421faf47effdceacd7b5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:47:03.184300 7950 raft_consensus.cc:482] T 00000000000000000000000000000000 P e3f4029a0c9b421faf47effdceacd7b5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:47:03.184378 7950 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e3f4029a0c9b421faf47effdceacd7b5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:47:03.185470 7950 raft_consensus.cc:504] T 00000000000000000000000000000000 P e3f4029a0c9b421faf47effdceacd7b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3f4029a0c9b421faf47effdceacd7b5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39425 } }
I20260619 14:47:03.185814 7950 leader_election.cc:302] T 00000000000000000000000000000000 P e3f4029a0c9b421faf47effdceacd7b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e3f4029a0c9b421faf47effdceacd7b5; no voters:
I20260619 14:47:03.186141 7950 leader_election.cc:288] T 00000000000000000000000000000000 P e3f4029a0c9b421faf47effdceacd7b5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:47:03.186343 7955 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e3f4029a0c9b421faf47effdceacd7b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:47:03.186676 7955 raft_consensus.cc:686] T 00000000000000000000000000000000 P e3f4029a0c9b421faf47effdceacd7b5 [term 1 LEADER]: Becoming Leader. State: Replica: e3f4029a0c9b421faf47effdceacd7b5, State: Running, Role: LEADER
I20260619 14:47:03.187040 7955 consensus_queue.cc:237] T 00000000000000000000000000000000 P e3f4029a0c9b421faf47effdceacd7b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3f4029a0c9b421faf47effdceacd7b5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39425 } }
I20260619 14:47:03.187623 7950 sys_catalog.cc:564] T 00000000000000000000000000000000 P e3f4029a0c9b421faf47effdceacd7b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:47:03.189004 7957 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3f4029a0c9b421faf47effdceacd7b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3f4029a0c9b421faf47effdceacd7b5. Latest consensus state: current_term: 1 leader_uuid: "e3f4029a0c9b421faf47effdceacd7b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3f4029a0c9b421faf47effdceacd7b5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39425 } } }
I20260619 14:47:03.189141 7957 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3f4029a0c9b421faf47effdceacd7b5 [sys.catalog]: This master's current role is: LEADER
I20260619 14:47:03.189536 7956 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3f4029a0c9b421faf47effdceacd7b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e3f4029a0c9b421faf47effdceacd7b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3f4029a0c9b421faf47effdceacd7b5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39425 } } }
I20260619 14:47:03.189646 7956 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3f4029a0c9b421faf47effdceacd7b5 [sys.catalog]: This master's current role is: LEADER
W20260619 14:47:03.192793 7970 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e3f4029a0c9b421faf47effdceacd7b5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:47:03.192905 7970 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:47:03.192996 7964 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:47:03.193778 7964 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:47:03.200277 7964 catalog_manager.cc:1347] Generated new cluster ID: a8ad7350866741b0b5eb4dbfb39a12b6
I20260619 14:47:03.200371 7964 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:47:03.221575 7964 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:47:03.223382 7964 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:47:03.237486 7964 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e3f4029a0c9b421faf47effdceacd7b5: Generated new TSK 0
I20260619 14:47:03.238750 7964 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880383132139-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880383132139-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260619 14:47:03.280951 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880383132139-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880383132139-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880383132139-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880383132139-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39425
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880383132139-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:47:03.405268 7976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:47:03.405561 7976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:47:03.405608 7976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:47:03.409824 7976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:47:03.409896 7976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:47:03.409937 7976 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 14:47:03.410023 7976 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 14:47:03.415608 7976 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880383132139-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880383132139-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880383132139-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880383132139-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:39425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7976
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880383132139-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:47:03.416877 7976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:47:03.417940 7976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:47:03.428381 7982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:47:03.428381 7984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:47:03.428438 7981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:47:03.428921 7976 server_base.cc:1034] running on GCE node
I20260619 14:47:03.429473 7976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:47:03.430424 7976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:47:03.431686 7976 hybrid_clock.cc:648] HybridClock initialized: now 1781880423431668 us; error 45 us; skew 500 ppm
Jun 19 14:47:03 dist-test-slave-2sb1 krb5kdc[7859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880423, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:47:03.435070 7976 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260619 14:47:03.436173 7976 webserver.cc:460] Webserver started at http://127.0.98.1:45851/ using document root <none> and password file <none>
I20260619 14:47:03.436863 7976 fs_manager.cc:359] Metadata directory not provided
I20260619 14:47:03.436962 7976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:47:03.437208 7976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:47:03.439388 7976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880383132139-392-0/minicluster-data/ts-0/data/instance:
uuid: "1d96bfd7d15b4150ab295c5af9efb461"
format_stamp: "Formatted at 2026-06-19 14:47:03 on dist-test-slave-2sb1"
server_key: "0f492a5fc74f069bb5a61fc5f8e0fe0c"
server_key_iv: "880e4b0ca6054596d28e9ffbf06d8e51"
server_key_version: "encryptionkey@0"
I20260619 14:47:03.440028 7976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880383132139-392-0/minicluster-data/ts-0/wal/instance:
uuid: "1d96bfd7d15b4150ab295c5af9efb461"
format_stamp: "Formatted at 2026-06-19 14:47:03 on dist-test-slave-2sb1"
server_key: "0f492a5fc74f069bb5a61fc5f8e0fe0c"
server_key_iv: "880e4b0ca6054596d28e9ffbf06d8e51"
server_key_version: "encryptionkey@0"
I20260619 14:47:03.444243 7976 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 14:47:03.447257 7991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:47:03.448953 7976 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:47:03.449107 7976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880383132139-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880383132139-392-0/minicluster-data/ts-0/wal
uuid: "1d96bfd7d15b4150ab295c5af9efb461"
format_stamp: "Formatted at 2026-06-19 14:47:03 on dist-test-slave-2sb1"
server_key: "0f492a5fc74f069bb5a61fc5f8e0fe0c"
server_key_iv: "880e4b0ca6054596d28e9ffbf06d8e51"
server_key_version: "encryptionkey@0"
I20260619 14:47:03.449251 7976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880383132139-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880383132139-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880383132139-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:47:03.462743 7976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:47:03.466327 7976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:47:03.466552 7976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:47:03.468010 7976 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:47:03.468092 7976 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:47:03.468165 7976 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:47:03.468199 7976 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 14:47:03 dist-test-slave-2sb1 krb5kdc[7859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880423, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 14:47:03.479518 7976 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41003
I20260619 14:47:03.479597 8106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41003 every 8 connection(s)
I20260619 14:47:03.480463 7976 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880383132139-392-0/minicluster-data/ts-0/data/info.pb
I20260619 14:47:03.483669 7953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:03.469140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37965 (local address 127.0.98.62:39425)
0619 14:47:03.469287 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:03.469290 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:03.470391 (+ 1101us) server_negotiation.cc:407] Connection header received
0619 14:47:03.471613 (+ 1222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:03.471616 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:03.471681 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:03.471766 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:03.473561 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:03.474180 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:03.475529 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:03.475773 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:03.475892 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:03.479447 (+ 3555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:47:03.479470 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:47:03.479474 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:47:03.479509 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:47:03.481361 (+ 1852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:03.481864 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:03.481867 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:03.481868 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:03.481948 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:03.482366 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:03.482369 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:03.482370 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:03.482554 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:47:03.482633 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:03.483287 (+ 654us) server_negotiation.cc:299] Negotiation successful
0619 14:47:03.483423 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 14:47:03.484467 8000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:03.469655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39425 (local address 127.0.98.1:37965)
0619 14:47:03.470215 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:03.470259 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:03.471352 (+ 1093us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:03.471983 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:03.471992 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:03.472732 (+ 740us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:47:03.473384 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:03.473405 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:03.474309 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:03.474313 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:03.475383 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:03.475394 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:03.475947 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:03.475951 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:03.476163 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:03.477555 (+ 1392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:47:03.477583 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:47:03.479253 (+ 1670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:47:03.481472 (+ 2219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:47:03.481479 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:47:03.481493 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:47:03.481765 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:47:03.482072 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:47:03.482075 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:47:03.482077 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:47:03.482264 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:47:03.482764 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:47:03.482769 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:47:03.483006 (+ 237us) client_negotiation.cc:769] Sending connection context
0619 14:47:03.483184 (+ 178us) client_negotiation.cc:240] Negotiation successful
0619 14:47:03.483450 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":148,"threads_started":1}
I20260619 14:47:03.486112 8107 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39425
I20260619 14:47:03.486299 8107 heartbeater.cc:464] Registering TS with master...
I20260619 14:47:03.486982 8107 heartbeater.cc:511] Master 127.0.98.62:39425 requested a full tablet report, sending...
I20260619 14:47:03.489053 7900 ts_manager.cc:194] Registered new tserver with Master: 1d96bfd7d15b4150ab295c5af9efb461 (127.0.98.1:41003)
I20260619 14:47:03.489302 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 7976
I20260619 14:47:03.489462 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880383132139-392-0/minicluster-data/ts-0/wal/instance
I20260619 14:47:03.489817 392 external_mini_cluster.cc:1428] Setting key 25630075ed652cb19f8c35efd2cad426
I20260619 14:47:03.490754 7900 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:37965
I20260619 14:47:03.499398 8113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:03.488608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39425 (local address 127.0.98.1:54991)
0619 14:47:03.489113 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:03.489130 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:03.489319 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:03.489803 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:03.489805 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:03.490030 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:47:03.490166 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:03.490174 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:03.491385 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:03.491390 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:03.492209 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:03.492219 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:03.492794 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:03.492798 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:03.492963 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:03.493726 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:47:03.493746 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:47:03.494337 (+ 591us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:47:03.497503 (+ 3166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:47:03.497506 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:47:03.497508 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:47:03.497741 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:47:03.498140 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:47:03.498143 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:47:03.498144 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:47:03.498232 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:47:03.498826 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:47:03.498829 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:47:03.498925 (+ 96us) client_negotiation.cc:769] Sending connection context
0619 14:47:03.499031 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 14:47:03.499224 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":389,"mutex_wait_us":27,"thread_start_us":147,"threads_started":1}
I20260619 14:47:03.499516 7953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:03.488758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54991 (local address 127.0.98.62:39425)
0619 14:47:03.488901 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:03.488904 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:03.489216 (+ 312us) server_negotiation.cc:407] Connection header received
0619 14:47:03.489511 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:03.489517 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:03.489570 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:03.489641 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:03.490338 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:03.491019 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:03.492352 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:03.492612 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:03.492798 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:03.494646 (+ 1848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:47:03.494649 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:47:03.494651 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:47:03.494721 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:47:03.497345 (+ 2624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:03.497878 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:03.497884 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:03.497889 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:03.498004 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:03.498340 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:03.498346 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:03.498350 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:03.498680 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:47:03.498835 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:03.499135 (+ 300us) server_negotiation.cc:299] Negotiation successful
0619 14:47:03.499280 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880383132139-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880383132139-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260619 14:47:03.525025 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880383132139-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880383132139-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880383132139-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880383132139-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39425
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880383132139-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:47:03.651438 8116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:47:03.651929 8116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:47:03.651985 8116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:47:03.657433 8116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:47:03.657670 8116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:47:03.657722 8116 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 14:47:03.657765 8116 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 14:47:03.663785 8116 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880383132139-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880383132139-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880383132139-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880383132139-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:39425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8116
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880383132139-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:47:03.664966 8116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:47:03.666026 8116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:47:03.677789 8124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:47:03.677985 8122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:47:03.678364 8121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:47:03.678678 8116 server_base.cc:1034] running on GCE node
I20260619 14:47:03.679322 8116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:47:03.680168 8116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:47:03.681396 8116 hybrid_clock.cc:648] HybridClock initialized: now 1781880423681350 us; error 57 us; skew 500 ppm
Jun 19 14:47:03 dist-test-slave-2sb1 krb5kdc[7859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880423, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:47:03.684998 8116 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260619 14:47:03.686354 8116 webserver.cc:460] Webserver started at http://127.0.98.2:41129/ using document root <none> and password file <none>
I20260619 14:47:03.687067 8116 fs_manager.cc:359] Metadata directory not provided
I20260619 14:47:03.687196 8116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:47:03.687433 8116 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:47:03.690146 8116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880383132139-392-0/minicluster-data/ts-1/data/instance:
uuid: "cddbd59e8c134b38a8fb7fa3b056ad9a"
format_stamp: "Formatted at 2026-06-19 14:47:03 on dist-test-slave-2sb1"
server_key: "b7ade13dc99e9e68afcfbde9b5cdf518"
server_key_iv: "e8b788f43bcc0d61304a013304ae7bf4"
server_key_version: "encryptionkey@0"
I20260619 14:47:03.690826 8116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880383132139-392-0/minicluster-data/ts-1/wal/instance:
uuid: "cddbd59e8c134b38a8fb7fa3b056ad9a"
format_stamp: "Formatted at 2026-06-19 14:47:03 on dist-test-slave-2sb1"
server_key: "b7ade13dc99e9e68afcfbde9b5cdf518"
server_key_iv: "e8b788f43bcc0d61304a013304ae7bf4"
server_key_version: "encryptionkey@0"
I20260619 14:47:03.695251 8116 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 14:47:03.698786 8131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:47:03.700313 8116 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 14:47:03.700445 8116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880383132139-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880383132139-392-0/minicluster-data/ts-1/wal
uuid: "cddbd59e8c134b38a8fb7fa3b056ad9a"
format_stamp: "Formatted at 2026-06-19 14:47:03 on dist-test-slave-2sb1"
server_key: "b7ade13dc99e9e68afcfbde9b5cdf518"
server_key_iv: "e8b788f43bcc0d61304a013304ae7bf4"
server_key_version: "encryptionkey@0"
I20260619 14:47:03.700631 8116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880383132139-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880383132139-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880383132139-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:47:03.723450 8116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:47:03.727133 8116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:47:03.727388 8116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:47:03.729020 8116 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:47:03.729080 8116 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:47:03.729122 8116 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:47:03.729138 8116 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 14:47:03 dist-test-slave-2sb1 krb5kdc[7859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880423, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 14:47:03.740697 8116 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43399
I20260619 14:47:03.740762 8246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43399 every 8 connection(s)
I20260619 14:47:03.741940 8116 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880383132139-392-0/minicluster-data/ts-1/data/info.pb
I20260619 14:47:03.743479 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 8116
I20260619 14:47:03.743608 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880383132139-392-0/minicluster-data/ts-1/wal/instance
I20260619 14:47:03.744103 392 external_mini_cluster.cc:1428] Setting key 9d87cb17e3b4b44285e597c39fe7df32
I20260619 14:47:03.745961 7953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:03.729971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51883 (local address 127.0.98.62:39425)
0619 14:47:03.730131 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:03.730136 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:03.731093 (+ 957us) server_negotiation.cc:407] Connection header received
0619 14:47:03.732431 (+ 1338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:03.732437 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:03.732513 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:03.732656 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:03.734339 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:03.734969 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:03.736359 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:03.736635 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:03.736805 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:03.740980 (+ 4175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:47:03.740985 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:47:03.740988 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:47:03.741019 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:47:03.742910 (+ 1891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:03.743447 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:03.743452 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:03.743456 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:03.743561 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:03.744146 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:03.744153 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:03.744156 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:03.744434 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:47:03.744568 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:03.745604 (+ 1036us) server_negotiation.cc:299] Negotiation successful
0619 14:47:03.745802 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 14:47:03.746840 8140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:03.730364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39425 (local address 127.0.98.2:51883)
0619 14:47:03.730884 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:03.730946 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:03.732137 (+ 1191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:03.732898 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:03.732905 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:03.733541 (+ 636us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:47:03.734166 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:03.734182 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:03.735116 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:03.735119 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:03.736213 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:03.736224 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:03.736861 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:03.736866 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:03.737140 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:03.738641 (+ 1501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:47:03.738666 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:47:03.740712 (+ 2046us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:47:03.743045 (+ 2333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:47:03.743050 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:47:03.743061 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:47:03.743318 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:47:03.743755 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:47:03.743759 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:47:03.743761 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:47:03.743920 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:47:03.744778 (+ 858us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:47:03.744786 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:47:03.745141 (+ 355us) client_negotiation.cc:769] Sending connection context
0619 14:47:03.745390 (+ 249us) client_negotiation.cc:240] Negotiation successful
0619 14:47:03.745801 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":124,"threads_started":1}
I20260619 14:47:03.748291 8247 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39425
I20260619 14:47:03.748576 8247 heartbeater.cc:464] Registering TS with master...
I20260619 14:47:03.749214 8247 heartbeater.cc:511] Master 127.0.98.62:39425 requested a full tablet report, sending...
I20260619 14:47:03.750449 7900 ts_manager.cc:194] Registered new tserver with Master: cddbd59e8c134b38a8fb7fa3b056ad9a (127.0.98.2:43399)
I20260619 14:47:03.751348 7900 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:51883
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
I20260619 14:47:03.760749 8254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:03.750976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39425 (local address 127.0.98.2:53785)
0619 14:47:03.751336 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:03.751353 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:03.751551 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:03.751863 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:03.751866 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:03.752150 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:47:03.752282 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:03.752290 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:03.753591 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:03.753595 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:03.754476 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:03.754486 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:03.755168 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:03.755171 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:03.755256 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:03.755845 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:47:03.755857 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:47:03.756419 (+ 562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:47:03.758420 (+ 2001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:47:03.758423 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:47:03.758425 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:47:03.758737 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:47:03.759070 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:47:03.759073 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:47:03.759075 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:47:03.759180 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:47:03.759948 (+ 768us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:47:03.759952 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:47:03.760068 (+ 116us) client_negotiation.cc:769] Sending connection context
0619 14:47:03.760200 (+ 132us) client_negotiation.cc:240] Negotiation successful
0619 14:47:03.760350 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":119,"threads_started":1}
I20260619 14:47:03.760754 7953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:03.751063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53785 (local address 127.0.98.62:39425)
0619 14:47:03.751199 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:03.751206 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:03.751426 (+ 220us) server_negotiation.cc:407] Connection header received
0619 14:47:03.751689 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:03.751693 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:03.751739 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:03.751845 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:03.752435 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:03.753336 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:03.754623 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:03.754997 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:03.755125 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:03.756609 (+ 1484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:47:03.756614 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:47:03.756621 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:47:03.756686 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:47:03.758290 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:03.758874 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:03.758876 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:03.758878 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:03.758951 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:03.759321 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:03.759363 (+ 42us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:03.759364 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:03.759785 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:47:03.759904 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:03.760303 (+ 399us) server_negotiation.cc:299] Negotiation successful
0619 14:47:03.760451 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880383132139-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880383132139-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260619 14:47:03.781816 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880383132139-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880383132139-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880383132139-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880383132139-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39425
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880383132139-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:47:03.905539 8256 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:47:03.905857 8256 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:47:03.905946 8256 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:47:03.910796 8256 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:47:03.910890 8256 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:47:03.910938 8256 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260619 14:47:03.911029 8256 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 14:47:03.917831 8256 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880383132139-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880383132139-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880383132139-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880383132139-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:39425
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8256
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880383132139-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:47:03.919081 8256 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:47:03.920213 8256 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:47:03.931047 8262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:47:03.931223 8264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:47:03.931105 8256 server_base.cc:1034] running on GCE node
W20260619 14:47:03.931209 8261 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:47:03.932049 8256 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:47:03.932981 8256 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:47:03.934211 8256 hybrid_clock.cc:648] HybridClock initialized: now 1781880423934192 us; error 39 us; skew 500 ppm
Jun 19 14:47:03 dist-test-slave-2sb1 krb5kdc[7859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880423, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:47:03.937465 8256 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260619 14:47:03.938597 8256 webserver.cc:460] Webserver started at http://127.0.98.3:36211/ using document root <none> and password file <none>
I20260619 14:47:03.939240 8256 fs_manager.cc:359] Metadata directory not provided
I20260619 14:47:03.939345 8256 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:47:03.939610 8256 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:47:03.942335 8256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880383132139-392-0/minicluster-data/ts-2/data/instance:
uuid: "273adc42fd3f43e69ef71827d279ab00"
format_stamp: "Formatted at 2026-06-19 14:47:03 on dist-test-slave-2sb1"
server_key: "3cbda645d5ca8266f1866f5614cf3e12"
server_key_iv: "2e518c9ac39fd3a99b0dc69991b39783"
server_key_version: "encryptionkey@0"
I20260619 14:47:03.943202 8256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880383132139-392-0/minicluster-data/ts-2/wal/instance:
uuid: "273adc42fd3f43e69ef71827d279ab00"
format_stamp: "Formatted at 2026-06-19 14:47:03 on dist-test-slave-2sb1"
server_key: "3cbda645d5ca8266f1866f5614cf3e12"
server_key_iv: "2e518c9ac39fd3a99b0dc69991b39783"
server_key_version: "encryptionkey@0"
I20260619 14:47:03.947853 8256 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 14:47:03.951165 8271 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:47:03.953347 8256 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260619 14:47:03.953480 8256 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880383132139-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880383132139-392-0/minicluster-data/ts-2/wal
uuid: "273adc42fd3f43e69ef71827d279ab00"
format_stamp: "Formatted at 2026-06-19 14:47:03 on dist-test-slave-2sb1"
server_key: "3cbda645d5ca8266f1866f5614cf3e12"
server_key_iv: "2e518c9ac39fd3a99b0dc69991b39783"
server_key_version: "encryptionkey@0"
I20260619 14:47:03.953652 8256 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880383132139-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880383132139-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880383132139-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:47:03.980119 8256 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:47:03.983680 8256 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:47:03.983866 8256 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:47:03.985598 8256 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:47:03.985726 8256 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:47:03.985782 8256 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:47:03.985836 8256 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 19 14:47:03 dist-test-slave-2sb1 krb5kdc[7859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880423, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 14:47:03.997236 8256 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43827
I20260619 14:47:03.997318 8386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43827 every 8 connection(s)
I20260619 14:47:03.998420 8256 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880383132139-392-0/minicluster-data/ts-2/data/info.pb
I20260619 14:47:03.999697 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 8256
I20260619 14:47:03.999842 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880383132139-392-0/minicluster-data/ts-2/wal/instance
I20260619 14:47:04.000285 392 external_mini_cluster.cc:1428] Setting key 16978c6fffe0a84cdbac457c3ee51438
I20260619 14:47:04.002584 7953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:03.986396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52589 (local address 127.0.98.62:39425)
0619 14:47:03.986512 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:03.986516 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:03.987720 (+ 1204us) server_negotiation.cc:407] Connection header received
0619 14:47:03.989052 (+ 1332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:03.989055 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:03.989118 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:03.989194 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:03.990820 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:03.991725 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:03.993653 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:03.994067 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:03.994190 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:03.997431 (+ 3241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:47:03.997444 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:47:03.997447 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:47:03.997479 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:47:03.999473 (+ 1994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:04.000202 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:04.000206 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:04.000208 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:04.000304 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:04.001024 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:04.001027 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:04.001028 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:04.001337 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:47:04.001620 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:04.002264 (+ 644us) server_negotiation.cc:299] Negotiation successful
0619 14:47:04.002398 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 14:47:04.003445 8280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:03.986991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39425 (local address 127.0.98.3:52589)
0619 14:47:03.987509 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:03.987596 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:03.988801 (+ 1205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:03.989391 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:03.989400 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:03.989907 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:47:03.990613 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:03.990633 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:03.991888 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:03.991893 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:03.993516 (+ 1623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:03.993528 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:03.994236 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:03.994239 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:03.994393 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:03.995084 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:47:03.995128 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:47:03.997193 (+ 2065us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:47:03.999610 (+ 2417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:47:03.999616 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:47:03.999631 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:47:04.000065 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:47:04.000480 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:47:04.000483 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:47:04.000486 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:47:04.000888 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:47:04.001488 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:47:04.001496 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:47:04.001993 (+ 497us) client_negotiation.cc:769] Sending connection context
0619 14:47:04.002172 (+ 179us) client_negotiation.cc:240] Negotiation successful
0619 14:47:04.002488 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":136,"threads_started":1}
I20260619 14:47:04.005004 8387 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39425
I20260619 14:47:04.005214 8387 heartbeater.cc:464] Registering TS with master...
I20260619 14:47:04.005909 8387 heartbeater.cc:511] Master 127.0.98.62:39425 requested a full tablet report, sending...
I20260619 14:47:04.007120 7901 ts_manager.cc:194] Registered new tserver with Master: 273adc42fd3f43e69ef71827d279ab00 (127.0.98.3:43827)
I20260619 14:47:04.007946 7901 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52589
I20260619 14:47:04.015390 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:47:04.017074 8393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:04.007444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39425 (local address 127.0.98.3:55061)
0619 14:47:04.007814 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:04.007830 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:04.007938 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:04.008355 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:04.008358 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:04.008601 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:47:04.008816 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:04.008824 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:04.009911 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:04.009915 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:04.010556 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:04.010568 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:04.011057 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:04.011060 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:04.011129 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:04.011876 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:47:04.011890 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:47:04.012344 (+ 454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:47:04.015105 (+ 2761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:47:04.015111 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:47:04.015115 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:47:04.015349 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:47:04.015848 (+ 499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:47:04.015850 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:47:04.015852 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:47:04.015942 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:47:04.016495 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:47:04.016498 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:47:04.016589 (+ 91us) client_negotiation.cc:769] Sending connection context
0619 14:47:04.016768 (+ 179us) client_negotiation.cc:240] Negotiation successful
0619 14:47:04.016911 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":147,"threads_started":1}
I20260619 14:47:04.017264 7953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:04.007574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55061 (local address 127.0.98.62:39425)
0619 14:47:04.007713 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:04.007718 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:04.007915 (+ 197us) server_negotiation.cc:407] Connection header received
0619 14:47:04.008154 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:04.008157 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:04.008227 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:04.008323 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:04.008964 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:04.009752 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:04.010682 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:04.010888 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:04.010972 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:04.012544 (+ 1572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:47:04.012549 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:47:04.012552 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:47:04.012584 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:47:04.014951 (+ 2367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:04.015440 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:04.015444 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:04.015448 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:04.015716 (+ 268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:04.016055 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:04.016059 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:04.016061 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:04.016343 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:47:04.016484 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:04.016841 (+ 357us) server_negotiation.cc:299] Negotiation successful
0619 14:47:04.016999 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
Jun 19 14:47:04 dist-test-slave-2sb1 krb5kdc[7859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880424, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 19 14:47:04 dist-test-slave-2sb1 krb5kdc[7859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:47:04 dist-test-slave-2sb1 krb5kdc[7859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 14:47:04.042691 7953 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:47:04.035395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48922 (local address 127.0.98.62:39425)
0619 14:47:04.035538 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:04.035542 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:04.035554 (+ 12us) server_negotiation.cc:407] Connection header received
0619 14:47:04.035594 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:04.035596 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:04.035643 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:04.035758 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:04.036529 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:04.037135 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:04.038037 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:04.038261 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:04.038377 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:04.042420 (+ 4043us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48922: BlockingRecv error: recv got EOF from 127.0.0.1:48922 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Jun 19 14:47:04 dist-test-slave-2sb1 krb5kdc[7859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880424, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 14:47:04.055711 7953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:04.044368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48926 (local address 127.0.98.62:39425)
0619 14:47:04.044537 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:04.044543 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:04.044751 (+ 208us) server_negotiation.cc:407] Connection header received
0619 14:47:04.044870 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:04.044874 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:04.044930 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:04.045028 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:04.045791 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:04.046371 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:04.047271 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:04.047512 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:04.047660 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:04.051298 (+ 3638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:47:04.051302 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:47:04.051304 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:47:04.051336 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:47:04.053296 (+ 1960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:04.054015 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:04.054019 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:04.054020 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:04.054297 (+ 277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:04.054716 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:04.054719 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:04.054721 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:04.054936 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:47:04.055051 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:04.055356 (+ 305us) server_negotiation.cc:299] Negotiation successful
0619 14:47:04.055470 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260619 14:47:04.059727 7901 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48926:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:47:04.063599 7901 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:47:04.080067 8412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:04.073919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60508 (local address 127.0.98.1:41003)
0619 14:47:04.074726 (+ 807us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:04.074736 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:04.074802 (+ 66us) server_negotiation.cc:407] Connection header received
0619 14:47:04.075030 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:04.075036 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:04.075242 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:04.075365 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:47:04.075802 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:04.077061 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:04.078573 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:04.079315 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:04.079447 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:04.079622 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:04.079812 (+ 190us) server_negotiation.cc:299] Negotiation successful
0619 14:47:04.079914 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":716,"thread_start_us":102,"threads_started":1}
I20260619 14:47:04.080425 8409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:04.073239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43399 (local address 127.0.0.1:45148)
0619 14:47:04.073944 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:04.073998 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:04.074350 (+ 352us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:04.075261 (+ 911us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:04.075268 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:04.075294 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:47:04.075467 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:04.075474 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:04.076775 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:04.076782 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:04.078336 (+ 1554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:04.078347 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:04.079836 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:04.079842 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:04.079960 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:04.079982 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 14:47:04.080047 (+ 65us) client_negotiation.cc:240] Negotiation successful
0619 14:47:04.080249 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"mutex_wait_us":19,"thread_start_us":110,"threads_started":1}
I20260619 14:47:04.080067 8413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:04.073626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41003 (local address 127.0.0.1:60508)
0619 14:47:04.074709 (+ 1083us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:04.074724 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:04.074873 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:04.075536 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:04.075541 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:04.075558 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:47:04.075672 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:04.075678 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:04.077189 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:04.077192 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:04.078444 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:04.078453 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:04.079449 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:04.079453 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:04.079574 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:04.079621 (+ 47us) client_negotiation.cc:769] Sending connection context
0619 14:47:04.079767 (+ 146us) client_negotiation.cc:240] Negotiation successful
0619 14:47:04.079906 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1020,"mutex_wait_us":1,"thread_start_us":106,"threads_started":1}
I20260619 14:47:04.081224 8410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:04.073416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45148 (local address 127.0.98.2:43399)
0619 14:47:04.074726 (+ 1310us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:04.074737 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:04.074762 (+ 25us) server_negotiation.cc:407] Connection header received
0619 14:47:04.074843 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:04.074848 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:04.075066 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:04.075591 (+ 525us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:47:04.075768 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:04.076585 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:04.078477 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:04.079526 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:04.079961 (+ 435us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:04.080237 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:04.080428 (+ 191us) server_negotiation.cc:299] Negotiation successful
0619 14:47:04.081078 (+ 650us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1174,"thread_start_us":113,"threads_started":1}
I20260619 14:47:04.081296 8411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:04.074355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43827 (local address 127.0.0.1:34512)
0619 14:47:04.075098 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:04.075111 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:04.075193 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:04.075689 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:04.075691 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:04.075708 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:47:04.075816 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:04.075822 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:04.077224 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:04.077228 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:04.078353 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:04.078364 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:04.079873 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:04.079876 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:04.080872 (+ 996us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:04.080899 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 14:47:04.080974 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 14:47:04.081155 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":673,"mutex_wait_us":80,"thread_start_us":89,"threads_started":1}
I20260619 14:47:04.081951 8414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:04.074556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34512 (local address 127.0.98.3:43827)
0619 14:47:04.074922 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:04.074949 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:04.075287 (+ 338us) server_negotiation.cc:407] Connection header received
0619 14:47:04.075364 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:04.075368 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:04.075528 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:04.075684 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:47:04.075940 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:04.077096 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:04.078518 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:04.079517 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:04.079648 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:04.079823 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:04.081672 (+ 1849us) server_negotiation.cc:299] Negotiation successful
0619 14:47:04.081780 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":110,"threads_started":1}
I20260619 14:47:04.083809 8041 tablet_service.cc:1467] Processing CreateTablet for tablet 846223b6eece4b89b716c49330e0c9cb (DEFAULT_TABLE table=test-table [id=12760713e0ab457093524546090c1727]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:47:04.084951 8041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 846223b6eece4b89b716c49330e0c9cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:47:04.084527 8321 tablet_service.cc:1467] Processing CreateTablet for tablet 846223b6eece4b89b716c49330e0c9cb (DEFAULT_TABLE table=test-table [id=12760713e0ab457093524546090c1727]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:47:04.084525 8181 tablet_service.cc:1467] Processing CreateTablet for tablet 846223b6eece4b89b716c49330e0c9cb (DEFAULT_TABLE table=test-table [id=12760713e0ab457093524546090c1727]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:47:04.085566 8321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 846223b6eece4b89b716c49330e0c9cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:47:04.085561 8181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 846223b6eece4b89b716c49330e0c9cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:47:04.092684 8415 tablet_bootstrap.cc:492] T 846223b6eece4b89b716c49330e0c9cb P 1d96bfd7d15b4150ab295c5af9efb461: Bootstrap starting.
I20260619 14:47:04.094333 8417 tablet_bootstrap.cc:492] T 846223b6eece4b89b716c49330e0c9cb P 273adc42fd3f43e69ef71827d279ab00: Bootstrap starting.
I20260619 14:47:04.095000 8415 tablet_bootstrap.cc:654] T 846223b6eece4b89b716c49330e0c9cb P 1d96bfd7d15b4150ab295c5af9efb461: Neither blocks nor log segments found. Creating new log.
I20260619 14:47:04.095758 8415 log.cc:824] T 846223b6eece4b89b716c49330e0c9cb P 1d96bfd7d15b4150ab295c5af9efb461: Log is configured to *not* fsync() on all Append() calls
I20260619 14:47:04.096108 8417 tablet_bootstrap.cc:654] T 846223b6eece4b89b716c49330e0c9cb P 273adc42fd3f43e69ef71827d279ab00: Neither blocks nor log segments found. Creating new log.
I20260619 14:47:04.097043 8417 log.cc:824] T 846223b6eece4b89b716c49330e0c9cb P 273adc42fd3f43e69ef71827d279ab00: Log is configured to *not* fsync() on all Append() calls
I20260619 14:47:04.097754 8416 tablet_bootstrap.cc:492] T 846223b6eece4b89b716c49330e0c9cb P cddbd59e8c134b38a8fb7fa3b056ad9a: Bootstrap starting.
I20260619 14:47:04.098354 8415 tablet_bootstrap.cc:492] T 846223b6eece4b89b716c49330e0c9cb P 1d96bfd7d15b4150ab295c5af9efb461: No bootstrap required, opened a new log
I20260619 14:47:04.098577 8415 ts_tablet_manager.cc:1397] T 846223b6eece4b89b716c49330e0c9cb P 1d96bfd7d15b4150ab295c5af9efb461: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 14:47:04.099172 8417 tablet_bootstrap.cc:492] T 846223b6eece4b89b716c49330e0c9cb P 273adc42fd3f43e69ef71827d279ab00: No bootstrap required, opened a new log
I20260619 14:47:04.099360 8417 ts_tablet_manager.cc:1397] T 846223b6eece4b89b716c49330e0c9cb P 273adc42fd3f43e69ef71827d279ab00: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 14:47:04.099741 8416 tablet_bootstrap.cc:654] T 846223b6eece4b89b716c49330e0c9cb P cddbd59e8c134b38a8fb7fa3b056ad9a: Neither blocks nor log segments found. Creating new log.
I20260619 14:47:04.100431 8416 log.cc:824] T 846223b6eece4b89b716c49330e0c9cb P cddbd59e8c134b38a8fb7fa3b056ad9a: Log is configured to *not* fsync() on all Append() calls
I20260619 14:47:04.102197 8416 tablet_bootstrap.cc:492] T 846223b6eece4b89b716c49330e0c9cb P cddbd59e8c134b38a8fb7fa3b056ad9a: No bootstrap required, opened a new log
I20260619 14:47:04.102401 8416 ts_tablet_manager.cc:1397] T 846223b6eece4b89b716c49330e0c9cb P cddbd59e8c134b38a8fb7fa3b056ad9a: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 14:47:04.103415 8415 raft_consensus.cc:348] T 846223b6eece4b89b716c49330e0c9cb P 1d96bfd7d15b4150ab295c5af9efb461 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cddbd59e8c134b38a8fb7fa3b056ad9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43399 } } peers { permanent_uuid: "273adc42fd3f43e69ef71827d279ab00" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43827 } } peers { permanent_uuid: "1d96bfd7d15b4150ab295c5af9efb461" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41003 } }
I20260619 14:47:04.103415 8417 raft_consensus.cc:348] T 846223b6eece4b89b716c49330e0c9cb P 273adc42fd3f43e69ef71827d279ab00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cddbd59e8c134b38a8fb7fa3b056ad9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43399 } } peers { permanent_uuid: "273adc42fd3f43e69ef71827d279ab00" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43827 } } peers { permanent_uuid: "1d96bfd7d15b4150ab295c5af9efb461" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41003 } }
I20260619 14:47:04.103703 8417 raft_consensus.cc:374] T 846223b6eece4b89b716c49330e0c9cb P 273adc42fd3f43e69ef71827d279ab00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:47:04.103703 8415 raft_consensus.cc:374] T 846223b6eece4b89b716c49330e0c9cb P 1d96bfd7d15b4150ab295c5af9efb461 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:47:04.103756 8417 raft_consensus.cc:729] T 846223b6eece4b89b716c49330e0c9cb P 273adc42fd3f43e69ef71827d279ab00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 273adc42fd3f43e69ef71827d279ab00, State: Initialized, Role: FOLLOWER
I20260619 14:47:04.103756 8415 raft_consensus.cc:729] T 846223b6eece4b89b716c49330e0c9cb P 1d96bfd7d15b4150ab295c5af9efb461 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d96bfd7d15b4150ab295c5af9efb461, State: Initialized, Role: FOLLOWER
I20260619 14:47:04.104318 8417 consensus_queue.cc:260] T 846223b6eece4b89b716c49330e0c9cb P 273adc42fd3f43e69ef71827d279ab00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cddbd59e8c134b38a8fb7fa3b056ad9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43399 } } peers { permanent_uuid: "273adc42fd3f43e69ef71827d279ab00" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43827 } } peers { permanent_uuid: "1d96bfd7d15b4150ab295c5af9efb461" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41003 } }
I20260619 14:47:04.104523 8415 consensus_queue.cc:260] T 846223b6eece4b89b716c49330e0c9cb P 1d96bfd7d15b4150ab295c5af9efb461 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cddbd59e8c134b38a8fb7fa3b056ad9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43399 } } peers { permanent_uuid: "273adc42fd3f43e69ef71827d279ab00" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43827 } } peers { permanent_uuid: "1d96bfd7d15b4150ab295c5af9efb461" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41003 } }
I20260619 14:47:04.105125 8387 heartbeater.cc:503] Master 127.0.98.62:39425 was elected leader, sending a full tablet report...
I20260619 14:47:04.105511 8415 ts_tablet_manager.cc:1428] T 846223b6eece4b89b716c49330e0c9cb P 1d96bfd7d15b4150ab295c5af9efb461: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260619 14:47:04.105511 8417 ts_tablet_manager.cc:1428] T 846223b6eece4b89b716c49330e0c9cb P 273adc42fd3f43e69ef71827d279ab00: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260619 14:47:04.105921 8107 heartbeater.cc:503] Master 127.0.98.62:39425 was elected leader, sending a full tablet report...
I20260619 14:47:04.106410 8416 raft_consensus.cc:348] T 846223b6eece4b89b716c49330e0c9cb P cddbd59e8c134b38a8fb7fa3b056ad9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cddbd59e8c134b38a8fb7fa3b056ad9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43399 } } peers { permanent_uuid: "273adc42fd3f43e69ef71827d279ab00" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43827 } } peers { permanent_uuid: "1d96bfd7d15b4150ab295c5af9efb461" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41003 } }
I20260619 14:47:04.106844 8416 raft_consensus.cc:374] T 846223b6eece4b89b716c49330e0c9cb P cddbd59e8c134b38a8fb7fa3b056ad9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:47:04.106907 8416 raft_consensus.cc:729] T 846223b6eece4b89b716c49330e0c9cb P cddbd59e8c134b38a8fb7fa3b056ad9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cddbd59e8c134b38a8fb7fa3b056ad9a, State: Initialized, Role: FOLLOWER
I20260619 14:47:04.107327 8416 consensus_queue.cc:260] T 846223b6eece4b89b716c49330e0c9cb P cddbd59e8c134b38a8fb7fa3b056ad9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cddbd59e8c134b38a8fb7fa3b056ad9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43399 } } peers { permanent_uuid: "273adc42fd3f43e69ef71827d279ab00" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43827 } } peers { permanent_uuid: "1d96bfd7d15b4150ab295c5af9efb461" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41003 } }
I20260619 14:47:04.108028 8247 heartbeater.cc:503] Master 127.0.98.62:39425 was elected leader, sending a full tablet report...
I20260619 14:47:04.108381 8416 ts_tablet_manager.cc:1428] T 846223b6eece4b89b716c49330e0c9cb P cddbd59e8c134b38a8fb7fa3b056ad9a: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 14:47:04.184028 8423 raft_consensus.cc:482] T 846223b6eece4b89b716c49330e0c9cb P cddbd59e8c134b38a8fb7fa3b056ad9a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:47:04.184320 8423 raft_consensus.cc:504] T 846223b6eece4b89b716c49330e0c9cb P cddbd59e8c134b38a8fb7fa3b056ad9a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cddbd59e8c134b38a8fb7fa3b056ad9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43399 } } peers { permanent_uuid: "273adc42fd3f43e69ef71827d279ab00" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43827 } } peers { permanent_uuid: "1d96bfd7d15b4150ab295c5af9efb461" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41003 } }
I20260619 14:47:04.185636 8423 leader_election.cc:288] T 846223b6eece4b89b716c49330e0c9cb P cddbd59e8c134b38a8fb7fa3b056ad9a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 273adc42fd3f43e69ef71827d279ab00 (127.0.98.3:43827), 1d96bfd7d15b4150ab295c5af9efb461 (127.0.98.1:41003)
I20260619 14:47:04.190336 8414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:04.186009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46207 (local address 127.0.98.3:43827)
0619 14:47:04.186221 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:04.186228 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:04.186248 (+ 20us) server_negotiation.cc:407] Connection header received
0619 14:47:04.186397 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:04.186400 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:04.186444 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:04.186541 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:47:04.186969 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:04.187642 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:04.189092 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:04.189730 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:04.189844 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:04.189912 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:04.190170 (+ 258us) server_negotiation.cc:299] Negotiation successful
0619 14:47:04.190234 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
I20260619 14:47:04.190383 8140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:04.185855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43827 (local address 127.0.98.2:46207)
0619 14:47:04.186111 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:04.186127 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:04.186235 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:04.186695 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:04.186699 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:04.186740 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:47:04.186838 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:04.186843 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:04.187795 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:04.187798 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:04.188869 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:04.188883 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:04.189870 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:04.189873 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:04.190011 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:04.190047 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 14:47:04.190141 (+ 94us) client_negotiation.cc:240] Negotiation successful
0619 14:47:04.190221 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"mutex_wait_us":1,"spinlock_wait_cycles":9600}
I20260619 14:47:04.191267 8424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:04.186003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41003 (local address 127.0.98.2:42949)
0619 14:47:04.186456 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:04.186468 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:04.186762 (+ 294us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:04.187033 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:04.187036 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:04.187051 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:47:04.187174 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:04.187179 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:04.187989 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:04.187993 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:04.189869 (+ 1876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:04.189879 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:04.190895 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:04.190902 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:04.191000 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:04.191023 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 14:47:04.191094 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 14:47:04.191145 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"mutex_wait_us":86,"spinlock_wait_cycles":45824,"thread_start_us":140,"threads_started":1}
I20260619 14:47:04.191324 8412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:04.186123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42949 (local address 127.0.98.1:41003)
0619 14:47:04.186283 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:04.186289 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:04.186681 (+ 392us) server_negotiation.cc:407] Connection header received
0619 14:47:04.186874 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:04.186877 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:04.186931 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:04.187016 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:47:04.187295 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:04.187829 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:04.190003 (+ 2174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:04.190764 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:04.190863 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:04.190945 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:04.191139 (+ 194us) server_negotiation.cc:299] Negotiation successful
0619 14:47:04.191212 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260619 14:47:04.191299 8341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "846223b6eece4b89b716c49330e0c9cb" candidate_uuid: "cddbd59e8c134b38a8fb7fa3b056ad9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "273adc42fd3f43e69ef71827d279ab00" is_pre_election: true
I20260619 14:47:04.191642 8341 raft_consensus.cc:2454] T 846223b6eece4b89b716c49330e0c9cb P 273adc42fd3f43e69ef71827d279ab00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cddbd59e8c134b38a8fb7fa3b056ad9a in term 0.
I20260619 14:47:04.192039 8061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "846223b6eece4b89b716c49330e0c9cb" candidate_uuid: "cddbd59e8c134b38a8fb7fa3b056ad9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d96bfd7d15b4150ab295c5af9efb461" is_pre_election: true
I20260619 14:47:04.192231 8134 leader_election.cc:302] T 846223b6eece4b89b716c49330e0c9cb P cddbd59e8c134b38a8fb7fa3b056ad9a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 273adc42fd3f43e69ef71827d279ab00, cddbd59e8c134b38a8fb7fa3b056ad9a; no voters:
I20260619 14:47:04.192250 8061 raft_consensus.cc:2454] T 846223b6eece4b89b716c49330e0c9cb P 1d96bfd7d15b4150ab295c5af9efb461 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cddbd59e8c134b38a8fb7fa3b056ad9a in term 0.
I20260619 14:47:04.192531 8423 raft_consensus.cc:2781] T 846223b6eece4b89b716c49330e0c9cb P cddbd59e8c134b38a8fb7fa3b056ad9a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:47:04.192678 8423 raft_consensus.cc:482] T 846223b6eece4b89b716c49330e0c9cb P cddbd59e8c134b38a8fb7fa3b056ad9a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:47:04.192740 8423 raft_consensus.cc:3037] T 846223b6eece4b89b716c49330e0c9cb P cddbd59e8c134b38a8fb7fa3b056ad9a [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:47:04.193802 8423 raft_consensus.cc:504] T 846223b6eece4b89b716c49330e0c9cb P cddbd59e8c134b38a8fb7fa3b056ad9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cddbd59e8c134b38a8fb7fa3b056ad9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43399 } } peers { permanent_uuid: "273adc42fd3f43e69ef71827d279ab00" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43827 } } peers { permanent_uuid: "1d96bfd7d15b4150ab295c5af9efb461" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41003 } }
I20260619 14:47:04.194183 8423 leader_election.cc:288] T 846223b6eece4b89b716c49330e0c9cb P cddbd59e8c134b38a8fb7fa3b056ad9a [CANDIDATE]: Term 1 election: Requested vote from peers 273adc42fd3f43e69ef71827d279ab00 (127.0.98.3:43827), 1d96bfd7d15b4150ab295c5af9efb461 (127.0.98.1:41003)
I20260619 14:47:04.194566 8341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "846223b6eece4b89b716c49330e0c9cb" candidate_uuid: "cddbd59e8c134b38a8fb7fa3b056ad9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "273adc42fd3f43e69ef71827d279ab00"
I20260619 14:47:04.194654 8061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "846223b6eece4b89b716c49330e0c9cb" candidate_uuid: "cddbd59e8c134b38a8fb7fa3b056ad9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d96bfd7d15b4150ab295c5af9efb461"
I20260619 14:47:04.194710 8341 raft_consensus.cc:3037] T 846223b6eece4b89b716c49330e0c9cb P 273adc42fd3f43e69ef71827d279ab00 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:47:04.194839 8061 raft_consensus.cc:3037] T 846223b6eece4b89b716c49330e0c9cb P 1d96bfd7d15b4150ab295c5af9efb461 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:47:04.195930 8061 raft_consensus.cc:2454] T 846223b6eece4b89b716c49330e0c9cb P 1d96bfd7d15b4150ab295c5af9efb461 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cddbd59e8c134b38a8fb7fa3b056ad9a in term 1.
I20260619 14:47:04.195940 8341 raft_consensus.cc:2454] T 846223b6eece4b89b716c49330e0c9cb P 273adc42fd3f43e69ef71827d279ab00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cddbd59e8c134b38a8fb7fa3b056ad9a in term 1.
I20260619 14:47:04.196300 8135 leader_election.cc:302] T 846223b6eece4b89b716c49330e0c9cb P cddbd59e8c134b38a8fb7fa3b056ad9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d96bfd7d15b4150ab295c5af9efb461, cddbd59e8c134b38a8fb7fa3b056ad9a; no voters:
I20260619 14:47:04.196532 8423 raft_consensus.cc:2781] T 846223b6eece4b89b716c49330e0c9cb P cddbd59e8c134b38a8fb7fa3b056ad9a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:47:04.196825 8423 raft_consensus.cc:686] T 846223b6eece4b89b716c49330e0c9cb P cddbd59e8c134b38a8fb7fa3b056ad9a [term 1 LEADER]: Becoming Leader. State: Replica: cddbd59e8c134b38a8fb7fa3b056ad9a, State: Running, Role: LEADER
I20260619 14:47:04.197227 8423 consensus_queue.cc:237] T 846223b6eece4b89b716c49330e0c9cb P cddbd59e8c134b38a8fb7fa3b056ad9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cddbd59e8c134b38a8fb7fa3b056ad9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43399 } } peers { permanent_uuid: "273adc42fd3f43e69ef71827d279ab00" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43827 } } peers { permanent_uuid: "1d96bfd7d15b4150ab295c5af9efb461" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41003 } }
I20260619 14:47:04.200970 7901 catalog_manager.cc:5495] T 846223b6eece4b89b716c49330e0c9cb P cddbd59e8c134b38a8fb7fa3b056ad9a reported cstate change: term changed from 0 to 1, leader changed from <none> to cddbd59e8c134b38a8fb7fa3b056ad9a (127.0.98.2). New cstate: current_term: 1 leader_uuid: "cddbd59e8c134b38a8fb7fa3b056ad9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cddbd59e8c134b38a8fb7fa3b056ad9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43399 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "273adc42fd3f43e69ef71827d279ab00" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1d96bfd7d15b4150ab295c5af9efb461" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41003 } health_report { overall_health: UNKNOWN } } }
W20260619 14:47:04.232232 8108 tablet.cc:2369] T 846223b6eece4b89b716c49330e0c9cb P 1d96bfd7d15b4150ab295c5af9efb461: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:47:04.236294 8410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:04.232122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45162 (local address 127.0.98.2:43399)
0619 14:47:04.232320 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:04.232325 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:04.232345 (+ 20us) server_negotiation.cc:407] Connection header received
0619 14:47:04.232438 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:04.232441 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:04.232510 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:04.232681 (+ 171us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:47:04.232957 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:04.233657 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:04.234625 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:04.234841 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:04.234923 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:04.235257 (+ 334us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:47:04.235745 (+ 488us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:47:04.235839 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:04.235988 (+ 149us) server_negotiation.cc:299] Negotiation successful
0619 14:47:04.236076 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 14:47:04.242522 8061 raft_consensus.cc:1261] T 846223b6eece4b89b716c49330e0c9cb P 1d96bfd7d15b4150ab295c5af9efb461 [term 1 FOLLOWER]: Refusing update from remote peer cddbd59e8c134b38a8fb7fa3b056ad9a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:47:04.243115 8341 raft_consensus.cc:1261] T 846223b6eece4b89b716c49330e0c9cb P 273adc42fd3f43e69ef71827d279ab00 [term 1 FOLLOWER]: Refusing update from remote peer cddbd59e8c134b38a8fb7fa3b056ad9a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260619 14:47:04.243211 8248 tablet.cc:2369] T 846223b6eece4b89b716c49330e0c9cb P cddbd59e8c134b38a8fb7fa3b056ad9a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:47:04.243330 8423 consensus_queue.cc:1035] T 846223b6eece4b89b716c49330e0c9cb P cddbd59e8c134b38a8fb7fa3b056ad9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d96bfd7d15b4150ab295c5af9efb461" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:47:04.243784 8425 consensus_queue.cc:1035] T 846223b6eece4b89b716c49330e0c9cb P cddbd59e8c134b38a8fb7fa3b056ad9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "273adc42fd3f43e69ef71827d279ab00" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 14:47:04.249612 8388 tablet.cc:2369] T 846223b6eece4b89b716c49330e0c9cb P 273adc42fd3f43e69ef71827d279ab00: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:47:04.251796 8428 mvcc.cc:205] Tried to move back new op lower bound from 7298582217690222592 to 7298582217515081728. Current Snapshot: MvccSnapshot[applied={T|T < 7298582217690222592}]
I20260619 14:47:04.254129 8429 mvcc.cc:205] Tried to move back new op lower bound from 7298582217690222592 to 7298582217515081728. Current Snapshot: MvccSnapshot[applied={T|T < 7298582217690222592}]
I20260619 14:47:04.262727 7901 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48926:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:47:04.262944 7901 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48926:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:47:04.266039 7901 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e3f4029a0c9b421faf47effdceacd7b5: Sending DeleteTablet for 3 replicas of tablet 846223b6eece4b89b716c49330e0c9cb
I20260619 14:47:04.267112 8181 tablet_service.cc:1514] Processing DeleteTablet for tablet 846223b6eece4b89b716c49330e0c9cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:47:04 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45148
I20260619 14:47:04.267184 8321 tablet_service.cc:1514] Processing DeleteTablet for tablet 846223b6eece4b89b716c49330e0c9cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:47:04 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34512
I20260619 14:47:04.267621 8439 tablet_replica.cc:330] T 846223b6eece4b89b716c49330e0c9cb P cddbd59e8c134b38a8fb7fa3b056ad9a: stopping tablet replica
I20260619 14:47:04.267771 8440 tablet_replica.cc:330] T 846223b6eece4b89b716c49330e0c9cb P 273adc42fd3f43e69ef71827d279ab00: stopping tablet replica
I20260619 14:47:04.267822 8041 tablet_service.cc:1514] Processing DeleteTablet for tablet 846223b6eece4b89b716c49330e0c9cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:47:04 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60508
I20260619 14:47:04.268114 8440 raft_consensus.cc:2229] T 846223b6eece4b89b716c49330e0c9cb P 273adc42fd3f43e69ef71827d279ab00 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:47:04.268131 8439 raft_consensus.cc:2229] T 846223b6eece4b89b716c49330e0c9cb P cddbd59e8c134b38a8fb7fa3b056ad9a [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:47:04.268347 7901 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48926:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260619 14:47:04.268466 8440 raft_consensus.cc:2258] T 846223b6eece4b89b716c49330e0c9cb P 273adc42fd3f43e69ef71827d279ab00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:47:04.268857 8439 raft_consensus.cc:2258] T 846223b6eece4b89b716c49330e0c9cb P cddbd59e8c134b38a8fb7fa3b056ad9a [term 1 FOLLOWER]: Raft consensus is shut down!
W20260619 14:47:04.268991 7901 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:47:04.270303 8439 ts_tablet_manager.cc:1905] T 846223b6eece4b89b716c49330e0c9cb P cddbd59e8c134b38a8fb7fa3b056ad9a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:47:04.270303 8440 ts_tablet_manager.cc:1905] T 846223b6eece4b89b716c49330e0c9cb P 273adc42fd3f43e69ef71827d279ab00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:47:04.273209 8440 ts_tablet_manager.cc:1918] T 846223b6eece4b89b716c49330e0c9cb P 273adc42fd3f43e69ef71827d279ab00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:47:04.273334 8440 log.cc:1196] T 846223b6eece4b89b716c49330e0c9cb P 273adc42fd3f43e69ef71827d279ab00: Deleting WAL directory at /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880383132139-392-0/minicluster-data/ts-2/wal/wals/846223b6eece4b89b716c49330e0c9cb
I20260619 14:47:04.273458 8439 ts_tablet_manager.cc:1918] T 846223b6eece4b89b716c49330e0c9cb P cddbd59e8c134b38a8fb7fa3b056ad9a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:47:04.273641 8439 log.cc:1196] T 846223b6eece4b89b716c49330e0c9cb P cddbd59e8c134b38a8fb7fa3b056ad9a: Deleting WAL directory at /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880383132139-392-0/minicluster-data/ts-1/wal/wals/846223b6eece4b89b716c49330e0c9cb
I20260619 14:47:04.273654 8440 ts_tablet_manager.cc:1939] T 846223b6eece4b89b716c49330e0c9cb P 273adc42fd3f43e69ef71827d279ab00: Deleting consensus metadata
I20260619 14:47:04.274072 8439 ts_tablet_manager.cc:1939] T 846223b6eece4b89b716c49330e0c9cb P cddbd59e8c134b38a8fb7fa3b056ad9a: Deleting consensus metadata
I20260619 14:47:04.275027 7890 catalog_manager.cc:4841] TS 273adc42fd3f43e69ef71827d279ab00 (127.0.98.3:43827): tablet 846223b6eece4b89b716c49330e0c9cb (table test-table [id=12760713e0ab457093524546090c1727]) successfully deleted
I20260619 14:47:04.275213 8321 tablet_service.cc:1467] Processing CreateTablet for tablet b634b557920043f1ac447baa507c6141 (DEFAULT_TABLE table=test-table [id=ecb8c1241aed49ca8dcd3f8e416e0791]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:47:04.275516 8321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b634b557920043f1ac447baa507c6141. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:47:04.276894 7890 catalog_manager.cc:4841] TS cddbd59e8c134b38a8fb7fa3b056ad9a (127.0.98.2:43399): tablet 846223b6eece4b89b716c49330e0c9cb (table test-table [id=12760713e0ab457093524546090c1727]) successfully deleted
I20260619 14:47:04.277400 8041 tablet_service.cc:1467] Processing CreateTablet for tablet b634b557920043f1ac447baa507c6141 (DEFAULT_TABLE table=test-table [id=ecb8c1241aed49ca8dcd3f8e416e0791]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:47:04.277483 8441 tablet_replica.cc:330] T 846223b6eece4b89b716c49330e0c9cb P 1d96bfd7d15b4150ab295c5af9efb461: stopping tablet replica
I20260619 14:47:04.277715 8041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b634b557920043f1ac447baa507c6141. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:47:04.277817 8417 tablet_bootstrap.cc:492] T b634b557920043f1ac447baa507c6141 P 273adc42fd3f43e69ef71827d279ab00: Bootstrap starting.
I20260619 14:47:04.277895 8441 raft_consensus.cc:2229] T 846223b6eece4b89b716c49330e0c9cb P 1d96bfd7d15b4150ab295c5af9efb461 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:47:04.278236 8441 raft_consensus.cc:2258] T 846223b6eece4b89b716c49330e0c9cb P 1d96bfd7d15b4150ab295c5af9efb461 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:47:04.278774 8417 tablet_bootstrap.cc:654] T b634b557920043f1ac447baa507c6141 P 273adc42fd3f43e69ef71827d279ab00: Neither blocks nor log segments found. Creating new log.
I20260619 14:47:04.279935 8181 tablet_service.cc:1467] Processing CreateTablet for tablet b634b557920043f1ac447baa507c6141 (DEFAULT_TABLE table=test-table [id=ecb8c1241aed49ca8dcd3f8e416e0791]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:47:04.280236 8181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b634b557920043f1ac447baa507c6141. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:47:04.280536 8415 tablet_bootstrap.cc:492] T b634b557920043f1ac447baa507c6141 P 1d96bfd7d15b4150ab295c5af9efb461: Bootstrap starting.
I20260619 14:47:04.281545 8415 tablet_bootstrap.cc:654] T b634b557920043f1ac447baa507c6141 P 1d96bfd7d15b4150ab295c5af9efb461: Neither blocks nor log segments found. Creating new log.
I20260619 14:47:04.282811 8416 tablet_bootstrap.cc:492] T b634b557920043f1ac447baa507c6141 P cddbd59e8c134b38a8fb7fa3b056ad9a: Bootstrap starting.
I20260619 14:47:04.284229 8417 tablet_bootstrap.cc:492] T b634b557920043f1ac447baa507c6141 P 273adc42fd3f43e69ef71827d279ab00: No bootstrap required, opened a new log
I20260619 14:47:04.284255 8416 tablet_bootstrap.cc:654] T b634b557920043f1ac447baa507c6141 P cddbd59e8c134b38a8fb7fa3b056ad9a: Neither blocks nor log segments found. Creating new log.
I20260619 14:47:04.284322 8417 ts_tablet_manager.cc:1397] T b634b557920043f1ac447baa507c6141 P 273adc42fd3f43e69ef71827d279ab00: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20260619 14:47:04.284859 8417 raft_consensus.cc:348] T b634b557920043f1ac447baa507c6141 P 273adc42fd3f43e69ef71827d279ab00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d96bfd7d15b4150ab295c5af9efb461" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41003 } } peers { permanent_uuid: "cddbd59e8c134b38a8fb7fa3b056ad9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43399 } } peers { permanent_uuid: "273adc42fd3f43e69ef71827d279ab00" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43827 } }
I20260619 14:47:04.284986 8417 raft_consensus.cc:374] T b634b557920043f1ac447baa507c6141 P 273adc42fd3f43e69ef71827d279ab00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:47:04.285023 8417 raft_consensus.cc:729] T b634b557920043f1ac447baa507c6141 P 273adc42fd3f43e69ef71827d279ab00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 273adc42fd3f43e69ef71827d279ab00, State: Initialized, Role: FOLLOWER
I20260619 14:47:04.285017 8441 ts_tablet_manager.cc:1905] T 846223b6eece4b89b716c49330e0c9cb P 1d96bfd7d15b4150ab295c5af9efb461: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:47:04.285184 8417 consensus_queue.cc:260] T b634b557920043f1ac447baa507c6141 P 273adc42fd3f43e69ef71827d279ab00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d96bfd7d15b4150ab295c5af9efb461" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41003 } } peers { permanent_uuid: "cddbd59e8c134b38a8fb7fa3b056ad9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43399 } } peers { permanent_uuid: "273adc42fd3f43e69ef71827d279ab00" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43827 } }
I20260619 14:47:04.285470 8417 ts_tablet_manager.cc:1428] T b634b557920043f1ac447baa507c6141 P 273adc42fd3f43e69ef71827d279ab00: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 14:47:04.288534 8441 ts_tablet_manager.cc:1918] T 846223b6eece4b89b716c49330e0c9cb P 1d96bfd7d15b4150ab295c5af9efb461: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:47:04.288687 8441 log.cc:1196] T 846223b6eece4b89b716c49330e0c9cb P 1d96bfd7d15b4150ab295c5af9efb461: Deleting WAL directory at /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880383132139-392-0/minicluster-data/ts-0/wal/wals/846223b6eece4b89b716c49330e0c9cb
I20260619 14:47:04.289477 8441 ts_tablet_manager.cc:1939] T 846223b6eece4b89b716c49330e0c9cb P 1d96bfd7d15b4150ab295c5af9efb461: Deleting consensus metadata
I20260619 14:47:04.290807 7891 catalog_manager.cc:4841] TS 1d96bfd7d15b4150ab295c5af9efb461 (127.0.98.1:41003): tablet 846223b6eece4b89b716c49330e0c9cb (table test-table [id=12760713e0ab457093524546090c1727]) successfully deleted
I20260619 14:47:04.291409 8415 tablet_bootstrap.cc:492] T b634b557920043f1ac447baa507c6141 P 1d96bfd7d15b4150ab295c5af9efb461: No bootstrap required, opened a new log
I20260619 14:47:04.291549 8415 ts_tablet_manager.cc:1397] T b634b557920043f1ac447baa507c6141 P 1d96bfd7d15b4150ab295c5af9efb461: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20260619 14:47:04.291989 8415 raft_consensus.cc:348] T b634b557920043f1ac447baa507c6141 P 1d96bfd7d15b4150ab295c5af9efb461 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d96bfd7d15b4150ab295c5af9efb461" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41003 } } peers { permanent_uuid: "cddbd59e8c134b38a8fb7fa3b056ad9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43399 } } peers { permanent_uuid: "273adc42fd3f43e69ef71827d279ab00" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43827 } }
I20260619 14:47:04.292135 8415 raft_consensus.cc:374] T b634b557920043f1ac447baa507c6141 P 1d96bfd7d15b4150ab295c5af9efb461 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:47:04.292196 8415 raft_consensus.cc:729] T b634b557920043f1ac447baa507c6141 P 1d96bfd7d15b4150ab295c5af9efb461 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d96bfd7d15b4150ab295c5af9efb461, State: Initialized, Role: FOLLOWER
I20260619 14:47:04.292398 8415 consensus_queue.cc:260] T b634b557920043f1ac447baa507c6141 P 1d96bfd7d15b4150ab295c5af9efb461 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d96bfd7d15b4150ab295c5af9efb461" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41003 } } peers { permanent_uuid: "cddbd59e8c134b38a8fb7fa3b056ad9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43399 } } peers { permanent_uuid: "273adc42fd3f43e69ef71827d279ab00" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43827 } }
I20260619 14:47:04.292754 8415 ts_tablet_manager.cc:1428] T b634b557920043f1ac447baa507c6141 P 1d96bfd7d15b4150ab295c5af9efb461: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260619 14:47:04.297960 8416 tablet_bootstrap.cc:492] T b634b557920043f1ac447baa507c6141 P cddbd59e8c134b38a8fb7fa3b056ad9a: No bootstrap required, opened a new log
I20260619 14:47:04.298105 8416 ts_tablet_manager.cc:1397] T b634b557920043f1ac447baa507c6141 P cddbd59e8c134b38a8fb7fa3b056ad9a: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.000s
I20260619 14:47:04.298517 8416 raft_consensus.cc:348] T b634b557920043f1ac447baa507c6141 P cddbd59e8c134b38a8fb7fa3b056ad9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d96bfd7d15b4150ab295c5af9efb461" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41003 } } peers { permanent_uuid: "cddbd59e8c134b38a8fb7fa3b056ad9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43399 } } peers { permanent_uuid: "273adc42fd3f43e69ef71827d279ab00" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43827 } }
I20260619 14:47:04.298612 8416 raft_consensus.cc:374] T b634b557920043f1ac447baa507c6141 P cddbd59e8c134b38a8fb7fa3b056ad9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:47:04.298635 8416 raft_consensus.cc:729] T b634b557920043f1ac447baa507c6141 P cddbd59e8c134b38a8fb7fa3b056ad9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cddbd59e8c134b38a8fb7fa3b056ad9a, State: Initialized, Role: FOLLOWER
I20260619 14:47:04.298794 8416 consensus_queue.cc:260] T b634b557920043f1ac447baa507c6141 P cddbd59e8c134b38a8fb7fa3b056ad9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d96bfd7d15b4150ab295c5af9efb461" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41003 } } peers { permanent_uuid: "cddbd59e8c134b38a8fb7fa3b056ad9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43399 } } peers { permanent_uuid: "273adc42fd3f43e69ef71827d279ab00" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43827 } }
I20260619 14:47:04.299089 8416 ts_tablet_manager.cc:1428] T b634b557920043f1ac447baa507c6141 P cddbd59e8c134b38a8fb7fa3b056ad9a: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 14:47:04.608853 8433 raft_consensus.cc:482] T b634b557920043f1ac447baa507c6141 P cddbd59e8c134b38a8fb7fa3b056ad9a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:47:04.609050 8433 raft_consensus.cc:504] T b634b557920043f1ac447baa507c6141 P cddbd59e8c134b38a8fb7fa3b056ad9a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d96bfd7d15b4150ab295c5af9efb461" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41003 } } peers { permanent_uuid: "cddbd59e8c134b38a8fb7fa3b056ad9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43399 } } peers { permanent_uuid: "273adc42fd3f43e69ef71827d279ab00" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43827 } }
I20260619 14:47:04.609577 8433 leader_election.cc:288] T b634b557920043f1ac447baa507c6141 P cddbd59e8c134b38a8fb7fa3b056ad9a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d96bfd7d15b4150ab295c5af9efb461 (127.0.98.1:41003), 273adc42fd3f43e69ef71827d279ab00 (127.0.98.3:43827)
I20260619 14:47:04.610328 8341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b634b557920043f1ac447baa507c6141" candidate_uuid: "cddbd59e8c134b38a8fb7fa3b056ad9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "273adc42fd3f43e69ef71827d279ab00" is_pre_election: true
I20260619 14:47:04.610337 8061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b634b557920043f1ac447baa507c6141" candidate_uuid: "cddbd59e8c134b38a8fb7fa3b056ad9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d96bfd7d15b4150ab295c5af9efb461" is_pre_election: true
I20260619 14:47:04.610512 8341 raft_consensus.cc:2454] T b634b557920043f1ac447baa507c6141 P 273adc42fd3f43e69ef71827d279ab00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cddbd59e8c134b38a8fb7fa3b056ad9a in term 0.
I20260619 14:47:04.610536 8061 raft_consensus.cc:2454] T b634b557920043f1ac447baa507c6141 P 1d96bfd7d15b4150ab295c5af9efb461 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cddbd59e8c134b38a8fb7fa3b056ad9a in term 0.
I20260619 14:47:04.610899 8134 leader_election.cc:302] T b634b557920043f1ac447baa507c6141 P cddbd59e8c134b38a8fb7fa3b056ad9a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 273adc42fd3f43e69ef71827d279ab00, cddbd59e8c134b38a8fb7fa3b056ad9a; no voters:
I20260619 14:47:04.611151 8433 raft_consensus.cc:2781] T b634b557920043f1ac447baa507c6141 P cddbd59e8c134b38a8fb7fa3b056ad9a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:47:04.611229 8433 raft_consensus.cc:482] T b634b557920043f1ac447baa507c6141 P cddbd59e8c134b38a8fb7fa3b056ad9a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:47:04.611261 8433 raft_consensus.cc:3037] T b634b557920043f1ac447baa507c6141 P cddbd59e8c134b38a8fb7fa3b056ad9a [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:47:04.612490 8433 raft_consensus.cc:504] T b634b557920043f1ac447baa507c6141 P cddbd59e8c134b38a8fb7fa3b056ad9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d96bfd7d15b4150ab295c5af9efb461" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41003 } } peers { permanent_uuid: "cddbd59e8c134b38a8fb7fa3b056ad9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43399 } } peers { permanent_uuid: "273adc42fd3f43e69ef71827d279ab00" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43827 } }
I20260619 14:47:04.612982 8433 leader_election.cc:288] T b634b557920043f1ac447baa507c6141 P cddbd59e8c134b38a8fb7fa3b056ad9a [CANDIDATE]: Term 1 election: Requested vote from peers 1d96bfd7d15b4150ab295c5af9efb461 (127.0.98.1:41003), 273adc42fd3f43e69ef71827d279ab00 (127.0.98.3:43827)
I20260619 14:47:04.613678 8061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b634b557920043f1ac447baa507c6141" candidate_uuid: "cddbd59e8c134b38a8fb7fa3b056ad9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d96bfd7d15b4150ab295c5af9efb461"
I20260619 14:47:04.613782 8341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b634b557920043f1ac447baa507c6141" candidate_uuid: "cddbd59e8c134b38a8fb7fa3b056ad9a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "273adc42fd3f43e69ef71827d279ab00"
I20260619 14:47:04.613847 8061 raft_consensus.cc:3037] T b634b557920043f1ac447baa507c6141 P 1d96bfd7d15b4150ab295c5af9efb461 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:47:04.613940 8341 raft_consensus.cc:3037] T b634b557920043f1ac447baa507c6141 P 273adc42fd3f43e69ef71827d279ab00 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:47:04.614895 8341 raft_consensus.cc:2454] T b634b557920043f1ac447baa507c6141 P 273adc42fd3f43e69ef71827d279ab00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cddbd59e8c134b38a8fb7fa3b056ad9a in term 1.
I20260619 14:47:04.614912 8061 raft_consensus.cc:2454] T b634b557920043f1ac447baa507c6141 P 1d96bfd7d15b4150ab295c5af9efb461 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cddbd59e8c134b38a8fb7fa3b056ad9a in term 1.
I20260619 14:47:04.615240 8134 leader_election.cc:302] T b634b557920043f1ac447baa507c6141 P cddbd59e8c134b38a8fb7fa3b056ad9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 273adc42fd3f43e69ef71827d279ab00, cddbd59e8c134b38a8fb7fa3b056ad9a; no voters:
I20260619 14:47:04.615489 8433 raft_consensus.cc:2781] T b634b557920043f1ac447baa507c6141 P cddbd59e8c134b38a8fb7fa3b056ad9a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:47:04.615607 8433 raft_consensus.cc:686] T b634b557920043f1ac447baa507c6141 P cddbd59e8c134b38a8fb7fa3b056ad9a [term 1 LEADER]: Becoming Leader. State: Replica: cddbd59e8c134b38a8fb7fa3b056ad9a, State: Running, Role: LEADER
I20260619 14:47:04.615765 8433 consensus_queue.cc:237] T b634b557920043f1ac447baa507c6141 P cddbd59e8c134b38a8fb7fa3b056ad9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d96bfd7d15b4150ab295c5af9efb461" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41003 } } peers { permanent_uuid: "cddbd59e8c134b38a8fb7fa3b056ad9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43399 } } peers { permanent_uuid: "273adc42fd3f43e69ef71827d279ab00" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43827 } }
I20260619 14:47:04.617676 7901 catalog_manager.cc:5495] T b634b557920043f1ac447baa507c6141 P cddbd59e8c134b38a8fb7fa3b056ad9a reported cstate change: term changed from 0 to 1, leader changed from <none> to cddbd59e8c134b38a8fb7fa3b056ad9a (127.0.98.2). New cstate: current_term: 1 leader_uuid: "cddbd59e8c134b38a8fb7fa3b056ad9a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d96bfd7d15b4150ab295c5af9efb461" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cddbd59e8c134b38a8fb7fa3b056ad9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43399 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "273adc42fd3f43e69ef71827d279ab00" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43827 } health_report { overall_health: UNKNOWN } } }
Jun 19 14:47:04 dist-test-slave-2sb1 krb5kdc[7859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880423, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 14:47:04.684443 8451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:04.674643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39425 (local address 127.0.0.1:48930)
0619 14:47:04.675096 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:04.675130 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:04.675256 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:04.675725 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:04.675727 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:04.676133 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:47:04.676224 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:04.676231 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:04.677099 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:04.677104 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:04.677659 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:04.677666 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:04.678126 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:04.678129 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:04.678197 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:04.678858 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:47:04.678871 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:47:04.680562 (+ 1691us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:47:04.682730 (+ 2168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:47:04.682740 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:47:04.682747 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:47:04.683003 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:47:04.683311 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:47:04.683313 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:47:04.683315 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:47:04.683401 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:47:04.683937 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:47:04.683944 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:47:04.684059 (+ 115us) client_negotiation.cc:769] Sending connection context
0619 14:47:04.684162 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 14:47:04.684292 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":167,"threads_started":1}
I20260619 14:47:04.684582 8452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:04.674881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48930 (local address 127.0.98.62:39425)
0619 14:47:04.675231 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:04.675240 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:04.675256 (+ 16us) server_negotiation.cc:407] Connection header received
0619 14:47:04.675381 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:04.675385 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:04.675434 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:04.675534 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:04.676351 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:04.676972 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:04.677776 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:04.678006 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:04.678125 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:04.680791 (+ 2666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:47:04.680794 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:47:04.680796 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:47:04.680824 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:47:04.682472 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:04.683108 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:04.683111 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:04.683113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:04.683197 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:04.683500 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:04.683619 (+ 119us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:04.683620 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:04.683819 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:47:04.683902 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:04.684268 (+ 366us) server_negotiation.cc:299] Negotiation successful
0619 14:47:04.684379 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":104,"threads_started":1}
I20260619 14:47:04.689437 7901 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48930:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260619 14:47:04.690398 7901 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:47:04.696875 8041 tablet_service.cc:1467] Processing CreateTablet for tablet 4f53d8eba91541758468bc8bea9f1285 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7fa6f8c471e4400da32a5086f67d33cb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 14:47:04.697002 8181 tablet_service.cc:1467] Processing CreateTablet for tablet 4f53d8eba91541758468bc8bea9f1285 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7fa6f8c471e4400da32a5086f67d33cb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 14:47:04.697383 8041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f53d8eba91541758468bc8bea9f1285. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:47:04.697261 8321 tablet_service.cc:1467] Processing CreateTablet for tablet 4f53d8eba91541758468bc8bea9f1285 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7fa6f8c471e4400da32a5086f67d33cb]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260619 14:47:04.697512 8181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f53d8eba91541758468bc8bea9f1285. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:47:04.697611 8321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f53d8eba91541758468bc8bea9f1285. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:47:04.700510 8415 tablet_bootstrap.cc:492] T 4f53d8eba91541758468bc8bea9f1285 P 1d96bfd7d15b4150ab295c5af9efb461: Bootstrap starting.
I20260619 14:47:04.700500 8416 tablet_bootstrap.cc:492] T 4f53d8eba91541758468bc8bea9f1285 P cddbd59e8c134b38a8fb7fa3b056ad9a: Bootstrap starting.
I20260619 14:47:04.701491 8415 tablet_bootstrap.cc:654] T 4f53d8eba91541758468bc8bea9f1285 P 1d96bfd7d15b4150ab295c5af9efb461: Neither blocks nor log segments found. Creating new log.
I20260619 14:47:04.701598 8416 tablet_bootstrap.cc:654] T 4f53d8eba91541758468bc8bea9f1285 P cddbd59e8c134b38a8fb7fa3b056ad9a: Neither blocks nor log segments found. Creating new log.
I20260619 14:47:04.702473 8417 tablet_bootstrap.cc:492] T 4f53d8eba91541758468bc8bea9f1285 P 273adc42fd3f43e69ef71827d279ab00: Bootstrap starting.
I20260619 14:47:04.703125 8416 tablet_bootstrap.cc:492] T 4f53d8eba91541758468bc8bea9f1285 P cddbd59e8c134b38a8fb7fa3b056ad9a: No bootstrap required, opened a new log
I20260619 14:47:04.703251 8416 ts_tablet_manager.cc:1397] T 4f53d8eba91541758468bc8bea9f1285 P cddbd59e8c134b38a8fb7fa3b056ad9a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 14:47:04.703529 8417 tablet_bootstrap.cc:654] T 4f53d8eba91541758468bc8bea9f1285 P 273adc42fd3f43e69ef71827d279ab00: Neither blocks nor log segments found. Creating new log.
I20260619 14:47:04.703753 8416 raft_consensus.cc:348] T 4f53d8eba91541758468bc8bea9f1285 P cddbd59e8c134b38a8fb7fa3b056ad9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d96bfd7d15b4150ab295c5af9efb461" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41003 } } peers { permanent_uuid: "cddbd59e8c134b38a8fb7fa3b056ad9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43399 } } peers { permanent_uuid: "273adc42fd3f43e69ef71827d279ab00" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43827 } }
I20260619 14:47:04.703890 8416 raft_consensus.cc:374] T 4f53d8eba91541758468bc8bea9f1285 P cddbd59e8c134b38a8fb7fa3b056ad9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:47:04.703927 8416 raft_consensus.cc:729] T 4f53d8eba91541758468bc8bea9f1285 P cddbd59e8c134b38a8fb7fa3b056ad9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cddbd59e8c134b38a8fb7fa3b056ad9a, State: Initialized, Role: FOLLOWER
I20260619 14:47:04.704090 8416 consensus_queue.cc:260] T 4f53d8eba91541758468bc8bea9f1285 P cddbd59e8c134b38a8fb7fa3b056ad9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d96bfd7d15b4150ab295c5af9efb461" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41003 } } peers { permanent_uuid: "cddbd59e8c134b38a8fb7fa3b056ad9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43399 } } peers { permanent_uuid: "273adc42fd3f43e69ef71827d279ab00" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43827 } }
I20260619 14:47:04.704355 8433 tablet_replica.cc:439] T 4f53d8eba91541758468bc8bea9f1285 P cddbd59e8c134b38a8fb7fa3b056ad9a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d96bfd7d15b4150ab295c5af9efb461" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41003 } } peers { permanent_uuid: "cddbd59e8c134b38a8fb7fa3b056ad9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43399 } } peers { permanent_uuid: "273adc42fd3f43e69ef71827d279ab00" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43827 } } }
I20260619 14:47:04.704540 8433 tablet_replica.cc:442] T 4f53d8eba91541758468bc8bea9f1285 P cddbd59e8c134b38a8fb7fa3b056ad9a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:47:04.704594 8415 tablet_bootstrap.cc:492] T 4f53d8eba91541758468bc8bea9f1285 P 1d96bfd7d15b4150ab295c5af9efb461: No bootstrap required, opened a new log
I20260619 14:47:04.704701 8415 ts_tablet_manager.cc:1397] T 4f53d8eba91541758468bc8bea9f1285 P 1d96bfd7d15b4150ab295c5af9efb461: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260619 14:47:04.705005 8416 ts_tablet_manager.cc:1428] T 4f53d8eba91541758468bc8bea9f1285 P cddbd59e8c134b38a8fb7fa3b056ad9a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 14:47:04.705250 8415 raft_consensus.cc:348] T 4f53d8eba91541758468bc8bea9f1285 P 1d96bfd7d15b4150ab295c5af9efb461 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d96bfd7d15b4150ab295c5af9efb461" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41003 } } peers { permanent_uuid: "cddbd59e8c134b38a8fb7fa3b056ad9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43399 } } peers { permanent_uuid: "273adc42fd3f43e69ef71827d279ab00" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43827 } }
I20260619 14:47:04.705379 8415 raft_consensus.cc:374] T 4f53d8eba91541758468bc8bea9f1285 P 1d96bfd7d15b4150ab295c5af9efb461 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:47:04.705405 8415 raft_consensus.cc:729] T 4f53d8eba91541758468bc8bea9f1285 P 1d96bfd7d15b4150ab295c5af9efb461 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d96bfd7d15b4150ab295c5af9efb461, State: Initialized, Role: FOLLOWER
I20260619 14:47:04.705559 8415 consensus_queue.cc:260] T 4f53d8eba91541758468bc8bea9f1285 P 1d96bfd7d15b4150ab295c5af9efb461 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d96bfd7d15b4150ab295c5af9efb461" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41003 } } peers { permanent_uuid: "cddbd59e8c134b38a8fb7fa3b056ad9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43399 } } peers { permanent_uuid: "273adc42fd3f43e69ef71827d279ab00" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43827 } }
I20260619 14:47:04.705834 8415 ts_tablet_manager.cc:1428] T 4f53d8eba91541758468bc8bea9f1285 P 1d96bfd7d15b4150ab295c5af9efb461: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 14:47:04.705783 8422 tablet_replica.cc:439] T 4f53d8eba91541758468bc8bea9f1285 P 1d96bfd7d15b4150ab295c5af9efb461: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d96bfd7d15b4150ab295c5af9efb461" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41003 } } peers { permanent_uuid: "cddbd59e8c134b38a8fb7fa3b056ad9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43399 } } peers { permanent_uuid: "273adc42fd3f43e69ef71827d279ab00" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43827 } } }
I20260619 14:47:04.705937 8422 tablet_replica.cc:442] T 4f53d8eba91541758468bc8bea9f1285 P 1d96bfd7d15b4150ab295c5af9efb461: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:47:04.706441 8417 tablet_bootstrap.cc:492] T 4f53d8eba91541758468bc8bea9f1285 P 273adc42fd3f43e69ef71827d279ab00: No bootstrap required, opened a new log
I20260619 14:47:04.706542 8417 ts_tablet_manager.cc:1397] T 4f53d8eba91541758468bc8bea9f1285 P 273adc42fd3f43e69ef71827d279ab00: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260619 14:47:04.707027 8417 raft_consensus.cc:348] T 4f53d8eba91541758468bc8bea9f1285 P 273adc42fd3f43e69ef71827d279ab00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d96bfd7d15b4150ab295c5af9efb461" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41003 } } peers { permanent_uuid: "cddbd59e8c134b38a8fb7fa3b056ad9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43399 } } peers { permanent_uuid: "273adc42fd3f43e69ef71827d279ab00" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43827 } }
I20260619 14:47:04.707149 8417 raft_consensus.cc:374] T 4f53d8eba91541758468bc8bea9f1285 P 273adc42fd3f43e69ef71827d279ab00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:47:04.707202 8417 raft_consensus.cc:729] T 4f53d8eba91541758468bc8bea9f1285 P 273adc42fd3f43e69ef71827d279ab00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 273adc42fd3f43e69ef71827d279ab00, State: Initialized, Role: FOLLOWER
I20260619 14:47:04.707331 8417 consensus_queue.cc:260] T 4f53d8eba91541758468bc8bea9f1285 P 273adc42fd3f43e69ef71827d279ab00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d96bfd7d15b4150ab295c5af9efb461" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41003 } } peers { permanent_uuid: "cddbd59e8c134b38a8fb7fa3b056ad9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43399 } } peers { permanent_uuid: "273adc42fd3f43e69ef71827d279ab00" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43827 } }
I20260619 14:47:04.707620 8417 ts_tablet_manager.cc:1428] T 4f53d8eba91541758468bc8bea9f1285 P 273adc42fd3f43e69ef71827d279ab00: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:47:04.707563 8421 tablet_replica.cc:439] T 4f53d8eba91541758468bc8bea9f1285 P 273adc42fd3f43e69ef71827d279ab00: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d96bfd7d15b4150ab295c5af9efb461" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41003 } } peers { permanent_uuid: "cddbd59e8c134b38a8fb7fa3b056ad9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43399 } } peers { permanent_uuid: "273adc42fd3f43e69ef71827d279ab00" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43827 } } }
I20260619 14:47:04.707715 8421 tablet_replica.cc:442] T 4f53d8eba91541758468bc8bea9f1285 P 273adc42fd3f43e69ef71827d279ab00: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:47:04.795059 8421 raft_consensus.cc:482] T 4f53d8eba91541758468bc8bea9f1285 P 273adc42fd3f43e69ef71827d279ab00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:47:04.795262 8421 raft_consensus.cc:504] T 4f53d8eba91541758468bc8bea9f1285 P 273adc42fd3f43e69ef71827d279ab00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d96bfd7d15b4150ab295c5af9efb461" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41003 } } peers { permanent_uuid: "cddbd59e8c134b38a8fb7fa3b056ad9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43399 } } peers { permanent_uuid: "273adc42fd3f43e69ef71827d279ab00" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43827 } }
I20260619 14:47:04.796588 8421 leader_election.cc:288] T 4f53d8eba91541758468bc8bea9f1285 P 273adc42fd3f43e69ef71827d279ab00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d96bfd7d15b4150ab295c5af9efb461 (127.0.98.1:41003), cddbd59e8c134b38a8fb7fa3b056ad9a (127.0.98.2:43399)
I20260619 14:47:04.802330 8456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:04.796934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41003 (local address 127.0.98.3:58367)
0619 14:47:04.797343 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:04.797356 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:04.797502 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:04.797925 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:04.797928 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:04.797976 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:47:04.798181 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:04.798189 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:04.799465 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:04.799468 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:04.801035 (+ 1567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:04.801049 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:04.801850 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:04.801855 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:04.801960 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:04.801993 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 14:47:04.802080 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 14:47:04.802149 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"mutex_wait_us":276,"spinlock_wait_cycles":32640,"thread_start_us":73,"threads_started":1}
I20260619 14:47:04.802376 8459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:04.797084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58367 (local address 127.0.98.1:41003)
0619 14:47:04.797368 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:04.797372 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:04.797457 (+ 85us) server_negotiation.cc:407] Connection header received
0619 14:47:04.797712 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:04.797716 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:04.797765 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:04.797870 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:47:04.798344 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:04.799346 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:04.801211 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:04.801693 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:04.801788 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:04.801842 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:04.802113 (+ 271us) server_negotiation.cc:299] Negotiation successful
0619 14:47:04.802204 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":73,"threads_started":1}
I20260619 14:47:04.802556 8458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:04.797060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37345 (local address 127.0.98.2:43399)
0619 14:47:04.797364 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:04.797370 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:04.797579 (+ 209us) server_negotiation.cc:407] Connection header received
0619 14:47:04.797693 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:04.797696 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:04.797757 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:04.797870 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:47:04.798340 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:04.799250 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:04.801249 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:04.801829 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:04.801944 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:04.802006 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:04.802296 (+ 290us) server_negotiation.cc:299] Negotiation successful
0619 14:47:04.802393 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":62,"threads_started":1}
I20260619 14:47:04.802641 8457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:04.796934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43399 (local address 127.0.98.3:37345)
0619 14:47:04.797327 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:04.797344 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:04.797582 (+ 238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:04.797925 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:04.797928 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:04.797978 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:47:04.798187 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:04.798191 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:04.799389 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:04.799392 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:04.801039 (+ 1647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:04.801115 (+ 76us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:04.801969 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:04.801973 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:04.802153 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:04.802176 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 14:47:04.802257 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 14:47:04.802332 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"mutex_wait_us":131,"spinlock_wait_cycles":10752,"thread_start_us":104,"threads_started":1}
I20260619 14:47:04.803375 8061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f53d8eba91541758468bc8bea9f1285" candidate_uuid: "273adc42fd3f43e69ef71827d279ab00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d96bfd7d15b4150ab295c5af9efb461" is_pre_election: true
I20260619 14:47:04.803555 8061 raft_consensus.cc:2454] T 4f53d8eba91541758468bc8bea9f1285 P 1d96bfd7d15b4150ab295c5af9efb461 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 273adc42fd3f43e69ef71827d279ab00 in term 0.
I20260619 14:47:04.803499 8201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f53d8eba91541758468bc8bea9f1285" candidate_uuid: "273adc42fd3f43e69ef71827d279ab00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cddbd59e8c134b38a8fb7fa3b056ad9a" is_pre_election: true
I20260619 14:47:04.803723 8201 raft_consensus.cc:2454] T 4f53d8eba91541758468bc8bea9f1285 P cddbd59e8c134b38a8fb7fa3b056ad9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 273adc42fd3f43e69ef71827d279ab00 in term 0.
I20260619 14:47:04.804126 8275 leader_election.cc:302] T 4f53d8eba91541758468bc8bea9f1285 P 273adc42fd3f43e69ef71827d279ab00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d96bfd7d15b4150ab295c5af9efb461, 273adc42fd3f43e69ef71827d279ab00; no voters:
I20260619 14:47:04.804786 8421 raft_consensus.cc:2781] T 4f53d8eba91541758468bc8bea9f1285 P 273adc42fd3f43e69ef71827d279ab00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:47:04.804863 8421 raft_consensus.cc:482] T 4f53d8eba91541758468bc8bea9f1285 P 273adc42fd3f43e69ef71827d279ab00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:47:04.804919 8421 raft_consensus.cc:3037] T 4f53d8eba91541758468bc8bea9f1285 P 273adc42fd3f43e69ef71827d279ab00 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:47:04.806110 8421 raft_consensus.cc:504] T 4f53d8eba91541758468bc8bea9f1285 P 273adc42fd3f43e69ef71827d279ab00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d96bfd7d15b4150ab295c5af9efb461" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41003 } } peers { permanent_uuid: "cddbd59e8c134b38a8fb7fa3b056ad9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43399 } } peers { permanent_uuid: "273adc42fd3f43e69ef71827d279ab00" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43827 } }
I20260619 14:47:04.806524 8421 leader_election.cc:288] T 4f53d8eba91541758468bc8bea9f1285 P 273adc42fd3f43e69ef71827d279ab00 [CANDIDATE]: Term 1 election: Requested vote from peers 1d96bfd7d15b4150ab295c5af9efb461 (127.0.98.1:41003), cddbd59e8c134b38a8fb7fa3b056ad9a (127.0.98.2:43399)
I20260619 14:47:04.806924 8201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f53d8eba91541758468bc8bea9f1285" candidate_uuid: "273adc42fd3f43e69ef71827d279ab00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cddbd59e8c134b38a8fb7fa3b056ad9a"
I20260619 14:47:04.806959 8061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f53d8eba91541758468bc8bea9f1285" candidate_uuid: "273adc42fd3f43e69ef71827d279ab00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d96bfd7d15b4150ab295c5af9efb461"
I20260619 14:47:04.807068 8061 raft_consensus.cc:3037] T 4f53d8eba91541758468bc8bea9f1285 P 1d96bfd7d15b4150ab295c5af9efb461 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:47:04.807065 8201 raft_consensus.cc:3037] T 4f53d8eba91541758468bc8bea9f1285 P cddbd59e8c134b38a8fb7fa3b056ad9a [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:47:04.808241 8061 raft_consensus.cc:2454] T 4f53d8eba91541758468bc8bea9f1285 P 1d96bfd7d15b4150ab295c5af9efb461 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 273adc42fd3f43e69ef71827d279ab00 in term 1.
I20260619 14:47:04.808241 8201 raft_consensus.cc:2454] T 4f53d8eba91541758468bc8bea9f1285 P cddbd59e8c134b38a8fb7fa3b056ad9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 273adc42fd3f43e69ef71827d279ab00 in term 1.
I20260619 14:47:04.808583 8275 leader_election.cc:302] T 4f53d8eba91541758468bc8bea9f1285 P 273adc42fd3f43e69ef71827d279ab00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d96bfd7d15b4150ab295c5af9efb461, 273adc42fd3f43e69ef71827d279ab00; no voters:
I20260619 14:47:04.808817 8421 raft_consensus.cc:2781] T 4f53d8eba91541758468bc8bea9f1285 P 273adc42fd3f43e69ef71827d279ab00 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:47:04.809013 8421 raft_consensus.cc:686] T 4f53d8eba91541758468bc8bea9f1285 P 273adc42fd3f43e69ef71827d279ab00 [term 1 LEADER]: Becoming Leader. State: Replica: 273adc42fd3f43e69ef71827d279ab00, State: Running, Role: LEADER
I20260619 14:47:04.809278 8421 consensus_queue.cc:237] T 4f53d8eba91541758468bc8bea9f1285 P 273adc42fd3f43e69ef71827d279ab00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d96bfd7d15b4150ab295c5af9efb461" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41003 } } peers { permanent_uuid: "cddbd59e8c134b38a8fb7fa3b056ad9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43399 } } peers { permanent_uuid: "273adc42fd3f43e69ef71827d279ab00" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43827 } }
I20260619 14:47:04.810490 8460 tablet_replica.cc:439] T 4f53d8eba91541758468bc8bea9f1285 P 273adc42fd3f43e69ef71827d279ab00: TxnStatusTablet state changed. Reason: New leader 273adc42fd3f43e69ef71827d279ab00. Latest consensus state: current_term: 1 leader_uuid: "273adc42fd3f43e69ef71827d279ab00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d96bfd7d15b4150ab295c5af9efb461" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41003 } } peers { permanent_uuid: "cddbd59e8c134b38a8fb7fa3b056ad9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43399 } } peers { permanent_uuid: "273adc42fd3f43e69ef71827d279ab00" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43827 } } }
I20260619 14:47:04.810606 8460 tablet_replica.cc:442] T 4f53d8eba91541758468bc8bea9f1285 P 273adc42fd3f43e69ef71827d279ab00: This TxnStatusTablet replica's current role is: LEADER
I20260619 14:47:04.810863 8462 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260619 14:47:04.811036 8462 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260619 14:47:04.811916 7904 catalog_manager.cc:5495] T 4f53d8eba91541758468bc8bea9f1285 P 273adc42fd3f43e69ef71827d279ab00 reported cstate change: term changed from 0 to 1, leader changed from <none> to 273adc42fd3f43e69ef71827d279ab00 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "273adc42fd3f43e69ef71827d279ab00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d96bfd7d15b4150ab295c5af9efb461" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cddbd59e8c134b38a8fb7fa3b056ad9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "273adc42fd3f43e69ef71827d279ab00" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43827 } health_report { overall_health: HEALTHY } } }
I20260619 14:47:04.836360 8465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:04.831692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34518 (local address 127.0.98.3:43827)
0619 14:47:04.832051 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:04.832092 (+ 41us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:04.832128 (+ 36us) server_negotiation.cc:407] Connection header received
0619 14:47:04.832215 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:04.832218 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:04.832273 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:04.832391 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:47:04.832693 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:04.833282 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:04.834309 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:04.834958 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:04.835097 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:04.835411 (+ 314us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:47:04.835895 (+ 484us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:47:04.836026 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:04.836126 (+ 100us) server_negotiation.cc:299] Negotiation successful
0619 14:47:04.836184 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":117,"threads_started":1}
I20260619 14:47:04.836344 8451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:04.831563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43827 (local address 127.0.0.1:34518)
0619 14:47:04.831780 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:04.831800 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:04.831925 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:04.832402 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:04.832405 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:04.832415 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 14:47:04.832526 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:04.832532 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:04.833414 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:04.833417 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:04.834176 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:04.834186 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:04.835093 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:04.835097 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:04.835192 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:04.835239 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 14:47:04.836000 (+ 761us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 14:47:04.836018 (+ 18us) client_negotiation.cc:769] Sending connection context
0619 14:47:04.836109 (+ 91us) client_negotiation.cc:240] Negotiation successful
0619 14:47:04.836224 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96}
I20260619 14:47:04.839691 8061 raft_consensus.cc:1261] T 4f53d8eba91541758468bc8bea9f1285 P 1d96bfd7d15b4150ab295c5af9efb461 [term 1 FOLLOWER]: Refusing update from remote peer 273adc42fd3f43e69ef71827d279ab00: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:47:04.840236 8201 raft_consensus.cc:1261] T 4f53d8eba91541758468bc8bea9f1285 P cddbd59e8c134b38a8fb7fa3b056ad9a [term 1 FOLLOWER]: Refusing update from remote peer 273adc42fd3f43e69ef71827d279ab00: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:47:04.840909 8421 consensus_queue.cc:1035] T 4f53d8eba91541758468bc8bea9f1285 P 273adc42fd3f43e69ef71827d279ab00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d96bfd7d15b4150ab295c5af9efb461" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:47:04.841126 8460 consensus_queue.cc:1035] T 4f53d8eba91541758468bc8bea9f1285 P 273adc42fd3f43e69ef71827d279ab00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cddbd59e8c134b38a8fb7fa3b056ad9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:47:04.843220 8433 tablet_replica.cc:439] T 4f53d8eba91541758468bc8bea9f1285 P cddbd59e8c134b38a8fb7fa3b056ad9a: TxnStatusTablet state changed. Reason: New leader 273adc42fd3f43e69ef71827d279ab00. Latest consensus state: current_term: 1 leader_uuid: "273adc42fd3f43e69ef71827d279ab00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d96bfd7d15b4150ab295c5af9efb461" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41003 } } peers { permanent_uuid: "cddbd59e8c134b38a8fb7fa3b056ad9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43399 } } peers { permanent_uuid: "273adc42fd3f43e69ef71827d279ab00" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43827 } } }
I20260619 14:47:04.843353 8433 tablet_replica.cc:442] T 4f53d8eba91541758468bc8bea9f1285 P cddbd59e8c134b38a8fb7fa3b056ad9a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:47:04.843611 8422 tablet_replica.cc:439] T 4f53d8eba91541758468bc8bea9f1285 P 1d96bfd7d15b4150ab295c5af9efb461: TxnStatusTablet state changed. Reason: New leader 273adc42fd3f43e69ef71827d279ab00. Latest consensus state: current_term: 1 leader_uuid: "273adc42fd3f43e69ef71827d279ab00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d96bfd7d15b4150ab295c5af9efb461" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41003 } } peers { permanent_uuid: "cddbd59e8c134b38a8fb7fa3b056ad9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43399 } } peers { permanent_uuid: "273adc42fd3f43e69ef71827d279ab00" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43827 } } }
I20260619 14:47:04.843747 8422 tablet_replica.cc:442] T 4f53d8eba91541758468bc8bea9f1285 P 1d96bfd7d15b4150ab295c5af9efb461: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:47:04.845832 8421 tablet_replica.cc:439] T 4f53d8eba91541758468bc8bea9f1285 P 273adc42fd3f43e69ef71827d279ab00: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "273adc42fd3f43e69ef71827d279ab00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d96bfd7d15b4150ab295c5af9efb461" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41003 } } peers { permanent_uuid: "cddbd59e8c134b38a8fb7fa3b056ad9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43399 } } peers { permanent_uuid: "273adc42fd3f43e69ef71827d279ab00" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43827 } } }
I20260619 14:47:04.845992 8421 tablet_replica.cc:442] T 4f53d8eba91541758468bc8bea9f1285 P 273adc42fd3f43e69ef71827d279ab00: This TxnStatusTablet replica's current role is: LEADER
I20260619 14:47:04.846155 8433 tablet_replica.cc:439] T 4f53d8eba91541758468bc8bea9f1285 P cddbd59e8c134b38a8fb7fa3b056ad9a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "273adc42fd3f43e69ef71827d279ab00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d96bfd7d15b4150ab295c5af9efb461" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41003 } } peers { permanent_uuid: "cddbd59e8c134b38a8fb7fa3b056ad9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43399 } } peers { permanent_uuid: "273adc42fd3f43e69ef71827d279ab00" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43827 } } }
I20260619 14:47:04.846266 8433 tablet_replica.cc:442] T 4f53d8eba91541758468bc8bea9f1285 P cddbd59e8c134b38a8fb7fa3b056ad9a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:47:04.853385 8421 tablet_replica.cc:439] T 4f53d8eba91541758468bc8bea9f1285 P 273adc42fd3f43e69ef71827d279ab00: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "273adc42fd3f43e69ef71827d279ab00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d96bfd7d15b4150ab295c5af9efb461" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41003 } } peers { permanent_uuid: "cddbd59e8c134b38a8fb7fa3b056ad9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43399 } } peers { permanent_uuid: "273adc42fd3f43e69ef71827d279ab00" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43827 } } }
I20260619 14:47:04.853544 8421 tablet_replica.cc:442] T 4f53d8eba91541758468bc8bea9f1285 P 273adc42fd3f43e69ef71827d279ab00: This TxnStatusTablet replica's current role is: LEADER
I20260619 14:47:04.854863 8422 tablet_replica.cc:439] T 4f53d8eba91541758468bc8bea9f1285 P 1d96bfd7d15b4150ab295c5af9efb461: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "273adc42fd3f43e69ef71827d279ab00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d96bfd7d15b4150ab295c5af9efb461" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41003 } } peers { permanent_uuid: "cddbd59e8c134b38a8fb7fa3b056ad9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43399 } } peers { permanent_uuid: "273adc42fd3f43e69ef71827d279ab00" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43827 } } }
I20260619 14:47:04.854995 8422 tablet_replica.cc:442] T 4f53d8eba91541758468bc8bea9f1285 P 1d96bfd7d15b4150ab295c5af9efb461: This TxnStatusTablet replica's current role is: FOLLOWER
I20260619 14:47:04.865298 8465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:04.861092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44951 (local address 127.0.98.3:43827)
0619 14:47:04.861255 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:04.861260 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:04.861444 (+ 184us) server_negotiation.cc:407] Connection header received
0619 14:47:04.861585 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:04.861588 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:04.861643 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:04.861715 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:47:04.861965 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:04.862508 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:04.863808 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:04.864084 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:04.864215 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:04.864805 (+ 590us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:47:04.864919 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:47:04.864998 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:04.865116 (+ 118us) server_negotiation.cc:299] Negotiation successful
0619 14:47:04.865170 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 14:47:04.865362 8477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:04.860997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43827 (local address 127.0.98.2:44951)
0619 14:47:04.861349 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:04.861365 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:04.861490 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:04.861743 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:04.861746 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:04.861754 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 14:47:04.861862 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:04.861868 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:04.862623 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:04.862628 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:04.863639 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:04.863652 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:04.864231 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:04.864235 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:04.864328 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:04.864393 (+ 65us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 14:47:04.865017 (+ 624us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 14:47:04.865025 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 14:47:04.865095 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 14:47:04.865197 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":137,"threads_started":1}
I20260619 14:47:04.873477 8341 raft_consensus.cc:1261] T b634b557920043f1ac447baa507c6141 P 273adc42fd3f43e69ef71827d279ab00 [term 1 FOLLOWER]: Refusing update from remote peer cddbd59e8c134b38a8fb7fa3b056ad9a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:47:04.873478 8061 raft_consensus.cc:1261] T b634b557920043f1ac447baa507c6141 P 1d96bfd7d15b4150ab295c5af9efb461 [term 1 FOLLOWER]: Refusing update from remote peer cddbd59e8c134b38a8fb7fa3b056ad9a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:47:04.873970 8425 consensus_queue.cc:1035] T b634b557920043f1ac447baa507c6141 P cddbd59e8c134b38a8fb7fa3b056ad9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d96bfd7d15b4150ab295c5af9efb461" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:47:04.874378 8433 consensus_queue.cc:1035] T b634b557920043f1ac447baa507c6141 P cddbd59e8c134b38a8fb7fa3b056ad9a [LEADER]: Connected to new peer: Peer: permanent_uuid: "273adc42fd3f43e69ef71827d279ab00" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:47:04.879979 8467 mvcc.cc:205] Tried to move back new op lower bound from 7298582220277559296 to 7298582219227463680. Current Snapshot: MvccSnapshot[applied={T|T < 7298582220277559296}]
I20260619 14:47:04.898211 8458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:04.893886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45837 (local address 127.0.98.2:43399)
0619 14:47:04.894009 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:04.894014 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:04.894282 (+ 268us) server_negotiation.cc:407] Connection header received
0619 14:47:04.894488 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:04.894491 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:04.894558 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:04.894629 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:47:04.894928 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:04.895726 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:04.897057 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:04.897273 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:04.897360 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:04.897722 (+ 362us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:47:04.897809 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:47:04.897895 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:04.897990 (+ 95us) server_negotiation.cc:299] Negotiation successful
0619 14:47:04.898101 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 14:47:04.898248 8483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:04.893736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43399 (local address 127.0.98.3:45837)
0619 14:47:04.894202 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:04.894217 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:04.894370 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:04.894664 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:04.894667 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:04.894675 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 14:47:04.894800 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:04.894807 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:04.895867 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:04.895870 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:04.896916 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:04.896928 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:04.897404 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:04.897408 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:04.897510 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:04.897568 (+ 58us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 14:47:04.897909 (+ 341us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 14:47:04.897916 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 14:47:04.897977 (+ 61us) client_negotiation.cc:240] Negotiation successful
0619 14:47:04.898065 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":141,"threads_started":1}
I20260619 14:47:04.924978 7904 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48926:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:47:04.925163 7904 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48926:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:47:04.927583 7904 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e3f4029a0c9b421faf47effdceacd7b5: Sending DeleteTablet for 3 replicas of tablet b634b557920043f1ac447baa507c6141
I20260619 14:47:04.928376 8181 tablet_service.cc:1514] Processing DeleteTablet for tablet b634b557920043f1ac447baa507c6141 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:47:04 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45148
I20260619 14:47:04.928426 8041 tablet_service.cc:1514] Processing DeleteTablet for tablet b634b557920043f1ac447baa507c6141 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:47:04 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60508
I20260619 14:47:04.928747 8321 tablet_service.cc:1514] Processing DeleteTablet for tablet b634b557920043f1ac447baa507c6141 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:47:04 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34512
I20260619 14:47:04.928857 8486 tablet_replica.cc:330] T b634b557920043f1ac447baa507c6141 P cddbd59e8c134b38a8fb7fa3b056ad9a: stopping tablet replica
I20260619 14:47:04.929003 8486 raft_consensus.cc:2229] T b634b557920043f1ac447baa507c6141 P cddbd59e8c134b38a8fb7fa3b056ad9a [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:47:04.929198 8488 tablet_replica.cc:330] T b634b557920043f1ac447baa507c6141 P 273adc42fd3f43e69ef71827d279ab00: stopping tablet replica
I20260619 14:47:04.929224 8486 raft_consensus.cc:2258] T b634b557920043f1ac447baa507c6141 P cddbd59e8c134b38a8fb7fa3b056ad9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:47:04.929384 8488 raft_consensus.cc:2229] T b634b557920043f1ac447baa507c6141 P 273adc42fd3f43e69ef71827d279ab00 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:47:04.929566 8488 raft_consensus.cc:2258] T b634b557920043f1ac447baa507c6141 P 273adc42fd3f43e69ef71827d279ab00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:47:04.929955 8486 ts_tablet_manager.cc:1905] T b634b557920043f1ac447baa507c6141 P cddbd59e8c134b38a8fb7fa3b056ad9a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:47:04.930644 8488 ts_tablet_manager.cc:1905] T b634b557920043f1ac447baa507c6141 P 273adc42fd3f43e69ef71827d279ab00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:47:04.933432 8487 tablet_replica.cc:330] T b634b557920043f1ac447baa507c6141 P 1d96bfd7d15b4150ab295c5af9efb461: stopping tablet replica
I20260619 14:47:04.933619 8487 raft_consensus.cc:2229] T b634b557920043f1ac447baa507c6141 P 1d96bfd7d15b4150ab295c5af9efb461 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:47:04.933790 8486 ts_tablet_manager.cc:1918] T b634b557920043f1ac447baa507c6141 P cddbd59e8c134b38a8fb7fa3b056ad9a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 14:47:04.933807 8487 raft_consensus.cc:2258] T b634b557920043f1ac447baa507c6141 P 1d96bfd7d15b4150ab295c5af9efb461 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:47:04.933887 8486 log.cc:1196] T b634b557920043f1ac447baa507c6141 P cddbd59e8c134b38a8fb7fa3b056ad9a: Deleting WAL directory at /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880383132139-392-0/minicluster-data/ts-1/wal/wals/b634b557920043f1ac447baa507c6141
I20260619 14:47:04.934240 8486 ts_tablet_manager.cc:1939] T b634b557920043f1ac447baa507c6141 P cddbd59e8c134b38a8fb7fa3b056ad9a: Deleting consensus metadata
I20260619 14:47:04.934646 8487 ts_tablet_manager.cc:1905] T b634b557920043f1ac447baa507c6141 P 1d96bfd7d15b4150ab295c5af9efb461: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:47:04.935038 8488 ts_tablet_manager.cc:1918] T b634b557920043f1ac447baa507c6141 P 273adc42fd3f43e69ef71827d279ab00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 14:47:04.935338 8488 log.cc:1196] T b634b557920043f1ac447baa507c6141 P 273adc42fd3f43e69ef71827d279ab00: Deleting WAL directory at /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880383132139-392-0/minicluster-data/ts-2/wal/wals/b634b557920043f1ac447baa507c6141
I20260619 14:47:04.935921 8488 ts_tablet_manager.cc:1939] T b634b557920043f1ac447baa507c6141 P 273adc42fd3f43e69ef71827d279ab00: Deleting consensus metadata
I20260619 14:47:04.936966 7890 catalog_manager.cc:4841] TS 273adc42fd3f43e69ef71827d279ab00 (127.0.98.3:43827): tablet b634b557920043f1ac447baa507c6141 (table test-table [id=ecb8c1241aed49ca8dcd3f8e416e0791]) successfully deleted
I20260619 14:47:04.937879 7890 catalog_manager.cc:4841] TS cddbd59e8c134b38a8fb7fa3b056ad9a (127.0.98.2:43399): tablet b634b557920043f1ac447baa507c6141 (table test-table [id=ecb8c1241aed49ca8dcd3f8e416e0791]) successfully deleted
I20260619 14:47:04.938803 8487 ts_tablet_manager.cc:1918] T b634b557920043f1ac447baa507c6141 P 1d96bfd7d15b4150ab295c5af9efb461: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 14:47:04.938903 8487 log.cc:1196] T b634b557920043f1ac447baa507c6141 P 1d96bfd7d15b4150ab295c5af9efb461: Deleting WAL directory at /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880383132139-392-0/minicluster-data/ts-0/wal/wals/b634b557920043f1ac447baa507c6141
I20260619 14:47:04.939229 8487 ts_tablet_manager.cc:1939] T b634b557920043f1ac447baa507c6141 P 1d96bfd7d15b4150ab295c5af9efb461: Deleting consensus metadata
I20260619 14:47:04.940232 7891 catalog_manager.cc:4841] TS 1d96bfd7d15b4150ab295c5af9efb461 (127.0.98.1:41003): tablet b634b557920043f1ac447baa507c6141 (table test-table [id=ecb8c1241aed49ca8dcd3f8e416e0791]) successfully deleted
Jun 19 14:47:04 dist-test-slave-2sb1 krb5kdc[7859](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880424, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 19 14:47:04 dist-test-slave-2sb1 krb5kdc[7859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:47:04 dist-test-slave-2sb1 krb5kdc[7859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 14:47:04.962989 8452 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:47:04.955349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48946 (local address 127.0.98.62:39425)
0619 14:47:04.955528 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:04.955532 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:04.955633 (+ 101us) server_negotiation.cc:407] Connection header received
0619 14:47:04.955819 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:04.955823 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:04.955878 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:04.955966 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:04.956908 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:04.957522 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:04.958505 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:04.958863 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:04.958987 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:04.962851 (+ 3864us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48946: BlockingRecv error: recv got EOF from 127.0.0.1:48946 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Jun 19 14:47:04 dist-test-slave-2sb1 krb5kdc[7859](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880424, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260619 14:47:04.974401 8452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:04.965132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48958 (local address 127.0.98.62:39425)
0619 14:47:04.965517 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:04.965522 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:04.965751 (+ 229us) server_negotiation.cc:407] Connection header received
0619 14:47:04.965928 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:04.965931 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:04.965975 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:04.966064 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:04.966704 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:04.967225 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:04.968033 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:04.968253 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:04.968367 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:04.971145 (+ 2778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:47:04.971148 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:47:04.971150 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:47:04.971178 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:47:04.972755 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:04.973248 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:04.973251 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:04.973252 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:04.973329 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:04.973645 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:04.973647 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:04.973648 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:04.973835 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:47:04.973936 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:04.974172 (+ 236us) server_negotiation.cc:299] Negotiation successful
0619 14:47:04.974277 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295}
I20260619 14:47:04.977329 7904 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48958:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:47:04.977918 7904 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:47:04.983167 8321 tablet_service.cc:1467] Processing CreateTablet for tablet d602dc5ebae74a7db1c4dfcaf8d13612 (DEFAULT_TABLE table=test-table [id=d7eb752f2e564df8bf6c24c4ee95703c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:47:04.983232 8181 tablet_service.cc:1467] Processing CreateTablet for tablet d602dc5ebae74a7db1c4dfcaf8d13612 (DEFAULT_TABLE table=test-table [id=d7eb752f2e564df8bf6c24c4ee95703c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:47:04.983291 8041 tablet_service.cc:1467] Processing CreateTablet for tablet d602dc5ebae74a7db1c4dfcaf8d13612 (DEFAULT_TABLE table=test-table [id=d7eb752f2e564df8bf6c24c4ee95703c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:47:04.983515 8181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d602dc5ebae74a7db1c4dfcaf8d13612. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:47:04.983528 8321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d602dc5ebae74a7db1c4dfcaf8d13612. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:47:04.983563 8041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d602dc5ebae74a7db1c4dfcaf8d13612. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:47:04.986263 8417 tablet_bootstrap.cc:492] T d602dc5ebae74a7db1c4dfcaf8d13612 P 273adc42fd3f43e69ef71827d279ab00: Bootstrap starting.
I20260619 14:47:04.986289 8415 tablet_bootstrap.cc:492] T d602dc5ebae74a7db1c4dfcaf8d13612 P 1d96bfd7d15b4150ab295c5af9efb461: Bootstrap starting.
I20260619 14:47:04.987068 8416 tablet_bootstrap.cc:492] T d602dc5ebae74a7db1c4dfcaf8d13612 P cddbd59e8c134b38a8fb7fa3b056ad9a: Bootstrap starting.
I20260619 14:47:04.987329 8417 tablet_bootstrap.cc:654] T d602dc5ebae74a7db1c4dfcaf8d13612 P 273adc42fd3f43e69ef71827d279ab00: Neither blocks nor log segments found. Creating new log.
I20260619 14:47:04.987331 8415 tablet_bootstrap.cc:654] T d602dc5ebae74a7db1c4dfcaf8d13612 P 1d96bfd7d15b4150ab295c5af9efb461: Neither blocks nor log segments found. Creating new log.
I20260619 14:47:04.988086 8416 tablet_bootstrap.cc:654] T d602dc5ebae74a7db1c4dfcaf8d13612 P cddbd59e8c134b38a8fb7fa3b056ad9a: Neither blocks nor log segments found. Creating new log.
I20260619 14:47:04.989243 8415 tablet_bootstrap.cc:492] T d602dc5ebae74a7db1c4dfcaf8d13612 P 1d96bfd7d15b4150ab295c5af9efb461: No bootstrap required, opened a new log
I20260619 14:47:04.989346 8415 ts_tablet_manager.cc:1397] T d602dc5ebae74a7db1c4dfcaf8d13612 P 1d96bfd7d15b4150ab295c5af9efb461: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 14:47:04.989580 8416 tablet_bootstrap.cc:492] T d602dc5ebae74a7db1c4dfcaf8d13612 P cddbd59e8c134b38a8fb7fa3b056ad9a: No bootstrap required, opened a new log
I20260619 14:47:04.989698 8416 ts_tablet_manager.cc:1397] T d602dc5ebae74a7db1c4dfcaf8d13612 P cddbd59e8c134b38a8fb7fa3b056ad9a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 14:47:04.990074 8415 raft_consensus.cc:348] T d602dc5ebae74a7db1c4dfcaf8d13612 P 1d96bfd7d15b4150ab295c5af9efb461 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d96bfd7d15b4150ab295c5af9efb461" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41003 } } peers { permanent_uuid: "cddbd59e8c134b38a8fb7fa3b056ad9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43399 } } peers { permanent_uuid: "273adc42fd3f43e69ef71827d279ab00" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43827 } }
I20260619 14:47:04.990209 8415 raft_consensus.cc:374] T d602dc5ebae74a7db1c4dfcaf8d13612 P 1d96bfd7d15b4150ab295c5af9efb461 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:47:04.990239 8415 raft_consensus.cc:729] T d602dc5ebae74a7db1c4dfcaf8d13612 P 1d96bfd7d15b4150ab295c5af9efb461 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d96bfd7d15b4150ab295c5af9efb461, State: Initialized, Role: FOLLOWER
I20260619 14:47:04.990156 8416 raft_consensus.cc:348] T d602dc5ebae74a7db1c4dfcaf8d13612 P cddbd59e8c134b38a8fb7fa3b056ad9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d96bfd7d15b4150ab295c5af9efb461" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41003 } } peers { permanent_uuid: "cddbd59e8c134b38a8fb7fa3b056ad9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43399 } } peers { permanent_uuid: "273adc42fd3f43e69ef71827d279ab00" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43827 } }
I20260619 14:47:04.990293 8416 raft_consensus.cc:374] T d602dc5ebae74a7db1c4dfcaf8d13612 P cddbd59e8c134b38a8fb7fa3b056ad9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:47:04.990331 8416 raft_consensus.cc:729] T d602dc5ebae74a7db1c4dfcaf8d13612 P cddbd59e8c134b38a8fb7fa3b056ad9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cddbd59e8c134b38a8fb7fa3b056ad9a, State: Initialized, Role: FOLLOWER
I20260619 14:47:04.990404 8415 consensus_queue.cc:260] T d602dc5ebae74a7db1c4dfcaf8d13612 P 1d96bfd7d15b4150ab295c5af9efb461 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d96bfd7d15b4150ab295c5af9efb461" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41003 } } peers { permanent_uuid: "cddbd59e8c134b38a8fb7fa3b056ad9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43399 } } peers { permanent_uuid: "273adc42fd3f43e69ef71827d279ab00" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43827 } }
I20260619 14:47:04.990468 8416 consensus_queue.cc:260] T d602dc5ebae74a7db1c4dfcaf8d13612 P cddbd59e8c134b38a8fb7fa3b056ad9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d96bfd7d15b4150ab295c5af9efb461" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41003 } } peers { permanent_uuid: "cddbd59e8c134b38a8fb7fa3b056ad9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43399 } } peers { permanent_uuid: "273adc42fd3f43e69ef71827d279ab00" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43827 } }
I20260619 14:47:04.990746 8416 ts_tablet_manager.cc:1428] T d602dc5ebae74a7db1c4dfcaf8d13612 P cddbd59e8c134b38a8fb7fa3b056ad9a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:47:04.992786 8415 ts_tablet_manager.cc:1428] T d602dc5ebae74a7db1c4dfcaf8d13612 P 1d96bfd7d15b4150ab295c5af9efb461: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260619 14:47:04.995365 8417 tablet_bootstrap.cc:492] T d602dc5ebae74a7db1c4dfcaf8d13612 P 273adc42fd3f43e69ef71827d279ab00: No bootstrap required, opened a new log
I20260619 14:47:04.995494 8417 ts_tablet_manager.cc:1397] T d602dc5ebae74a7db1c4dfcaf8d13612 P 273adc42fd3f43e69ef71827d279ab00: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.000s
I20260619 14:47:04.995901 8417 raft_consensus.cc:348] T d602dc5ebae74a7db1c4dfcaf8d13612 P 273adc42fd3f43e69ef71827d279ab00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d96bfd7d15b4150ab295c5af9efb461" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41003 } } peers { permanent_uuid: "cddbd59e8c134b38a8fb7fa3b056ad9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43399 } } peers { permanent_uuid: "273adc42fd3f43e69ef71827d279ab00" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43827 } }
I20260619 14:47:04.996030 8417 raft_consensus.cc:374] T d602dc5ebae74a7db1c4dfcaf8d13612 P 273adc42fd3f43e69ef71827d279ab00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:47:04.996074 8417 raft_consensus.cc:729] T d602dc5ebae74a7db1c4dfcaf8d13612 P 273adc42fd3f43e69ef71827d279ab00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 273adc42fd3f43e69ef71827d279ab00, State: Initialized, Role: FOLLOWER
I20260619 14:47:04.996217 8417 consensus_queue.cc:260] T d602dc5ebae74a7db1c4dfcaf8d13612 P 273adc42fd3f43e69ef71827d279ab00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d96bfd7d15b4150ab295c5af9efb461" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41003 } } peers { permanent_uuid: "cddbd59e8c134b38a8fb7fa3b056ad9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43399 } } peers { permanent_uuid: "273adc42fd3f43e69ef71827d279ab00" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43827 } }
I20260619 14:47:04.996506 8417 ts_tablet_manager.cc:1428] T d602dc5ebae74a7db1c4dfcaf8d13612 P 273adc42fd3f43e69ef71827d279ab00: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 14:47:05.028066 8422 raft_consensus.cc:482] T d602dc5ebae74a7db1c4dfcaf8d13612 P 1d96bfd7d15b4150ab295c5af9efb461 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:47:05.028250 8422 raft_consensus.cc:504] T d602dc5ebae74a7db1c4dfcaf8d13612 P 1d96bfd7d15b4150ab295c5af9efb461 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d96bfd7d15b4150ab295c5af9efb461" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41003 } } peers { permanent_uuid: "cddbd59e8c134b38a8fb7fa3b056ad9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43399 } } peers { permanent_uuid: "273adc42fd3f43e69ef71827d279ab00" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43827 } }
I20260619 14:47:05.029425 8422 leader_election.cc:288] T d602dc5ebae74a7db1c4dfcaf8d13612 P 1d96bfd7d15b4150ab295c5af9efb461 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cddbd59e8c134b38a8fb7fa3b056ad9a (127.0.98.2:43399), 273adc42fd3f43e69ef71827d279ab00 (127.0.98.3:43827)
I20260619 14:47:05.034647 8504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:05.029831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43399 (local address 127.0.98.1:50965)
0619 14:47:05.030223 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:05.030237 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:05.030385 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:05.030862 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:05.030869 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:05.030898 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:47:05.031045 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:05.031053 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:05.031967 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:05.031970 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:05.033403 (+ 1433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:05.033414 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:05.034148 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:05.034152 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:05.034272 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:05.034298 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 14:47:05.034369 (+ 71us) client_negotiation.cc:240] Negotiation successful
0619 14:47:05.034446 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"mutex_wait_us":39,"thread_start_us":154,"threads_started":1}
I20260619 14:47:05.034814 8465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:05.030524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47007 (local address 127.0.98.3:43827)
0619 14:47:05.030678 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:05.030692 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:05.030849 (+ 157us) server_negotiation.cc:407] Connection header received
0619 14:47:05.030957 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:05.030960 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:05.031013 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:05.031187 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:47:05.031453 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:05.032206 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:05.033659 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:05.034138 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:05.034284 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:05.034351 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:05.034588 (+ 237us) server_negotiation.cc:299] Negotiation successful
0619 14:47:05.034683 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 14:47:05.034847 8505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:05.030384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43827 (local address 127.0.98.1:47007)
0619 14:47:05.030752 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:05.030768 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:05.030860 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:05.031199 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:05.031202 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:05.031221 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:47:05.031344 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:05.031349 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:05.032378 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:05.032380 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:05.033520 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:05.033531 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:05.034297 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:05.034300 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:05.034421 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:05.034451 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 14:47:05.034560 (+ 109us) client_negotiation.cc:240] Negotiation successful
0619 14:47:05.034647 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"mutex_wait_us":53,"thread_start_us":120,"threads_started":1}
I20260619 14:47:05.035182 8458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:05.030010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50965 (local address 127.0.98.2:43399)
0619 14:47:05.030255 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:05.030263 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:05.030308 (+ 45us) server_negotiation.cc:407] Connection header received
0619 14:47:05.030506 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:05.030508 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:05.030550 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:05.030816 (+ 266us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:47:05.031194 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:05.031842 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:05.033532 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:05.034000 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:05.034133 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:05.034200 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:05.034931 (+ 731us) server_negotiation.cc:299] Negotiation successful
0619 14:47:05.035054 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260619 14:47:05.035473 8341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d602dc5ebae74a7db1c4dfcaf8d13612" candidate_uuid: "1d96bfd7d15b4150ab295c5af9efb461" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "273adc42fd3f43e69ef71827d279ab00" is_pre_election: true
I20260619 14:47:05.035568 8201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d602dc5ebae74a7db1c4dfcaf8d13612" candidate_uuid: "1d96bfd7d15b4150ab295c5af9efb461" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cddbd59e8c134b38a8fb7fa3b056ad9a" is_pre_election: true
I20260619 14:47:05.035655 8341 raft_consensus.cc:2454] T d602dc5ebae74a7db1c4dfcaf8d13612 P 273adc42fd3f43e69ef71827d279ab00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d96bfd7d15b4150ab295c5af9efb461 in term 0.
I20260619 14:47:05.035749 8201 raft_consensus.cc:2454] T d602dc5ebae74a7db1c4dfcaf8d13612 P cddbd59e8c134b38a8fb7fa3b056ad9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d96bfd7d15b4150ab295c5af9efb461 in term 0.
I20260619 14:47:05.036084 7994 leader_election.cc:302] T d602dc5ebae74a7db1c4dfcaf8d13612 P 1d96bfd7d15b4150ab295c5af9efb461 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d96bfd7d15b4150ab295c5af9efb461, 273adc42fd3f43e69ef71827d279ab00; no voters:
I20260619 14:47:05.036423 8422 raft_consensus.cc:2781] T d602dc5ebae74a7db1c4dfcaf8d13612 P 1d96bfd7d15b4150ab295c5af9efb461 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:47:05.036499 8422 raft_consensus.cc:482] T d602dc5ebae74a7db1c4dfcaf8d13612 P 1d96bfd7d15b4150ab295c5af9efb461 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:47:05.036527 8422 raft_consensus.cc:3037] T d602dc5ebae74a7db1c4dfcaf8d13612 P 1d96bfd7d15b4150ab295c5af9efb461 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:47:05.037662 8422 raft_consensus.cc:504] T d602dc5ebae74a7db1c4dfcaf8d13612 P 1d96bfd7d15b4150ab295c5af9efb461 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d96bfd7d15b4150ab295c5af9efb461" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41003 } } peers { permanent_uuid: "cddbd59e8c134b38a8fb7fa3b056ad9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43399 } } peers { permanent_uuid: "273adc42fd3f43e69ef71827d279ab00" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43827 } }
I20260619 14:47:05.038017 8422 leader_election.cc:288] T d602dc5ebae74a7db1c4dfcaf8d13612 P 1d96bfd7d15b4150ab295c5af9efb461 [CANDIDATE]: Term 1 election: Requested vote from peers cddbd59e8c134b38a8fb7fa3b056ad9a (127.0.98.2:43399), 273adc42fd3f43e69ef71827d279ab00 (127.0.98.3:43827)
I20260619 14:47:05.038494 8201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d602dc5ebae74a7db1c4dfcaf8d13612" candidate_uuid: "1d96bfd7d15b4150ab295c5af9efb461" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cddbd59e8c134b38a8fb7fa3b056ad9a"
I20260619 14:47:05.038594 8341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d602dc5ebae74a7db1c4dfcaf8d13612" candidate_uuid: "1d96bfd7d15b4150ab295c5af9efb461" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "273adc42fd3f43e69ef71827d279ab00"
I20260619 14:47:05.038663 8201 raft_consensus.cc:3037] T d602dc5ebae74a7db1c4dfcaf8d13612 P cddbd59e8c134b38a8fb7fa3b056ad9a [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:47:05.038717 8341 raft_consensus.cc:3037] T d602dc5ebae74a7db1c4dfcaf8d13612 P 273adc42fd3f43e69ef71827d279ab00 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:47:05.039964 8201 raft_consensus.cc:2454] T d602dc5ebae74a7db1c4dfcaf8d13612 P cddbd59e8c134b38a8fb7fa3b056ad9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d96bfd7d15b4150ab295c5af9efb461 in term 1.
I20260619 14:47:05.039964 8341 raft_consensus.cc:2454] T d602dc5ebae74a7db1c4dfcaf8d13612 P 273adc42fd3f43e69ef71827d279ab00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d96bfd7d15b4150ab295c5af9efb461 in term 1.
I20260619 14:47:05.040313 7994 leader_election.cc:302] T d602dc5ebae74a7db1c4dfcaf8d13612 P 1d96bfd7d15b4150ab295c5af9efb461 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d96bfd7d15b4150ab295c5af9efb461, cddbd59e8c134b38a8fb7fa3b056ad9a; no voters:
I20260619 14:47:05.040522 8422 raft_consensus.cc:2781] T d602dc5ebae74a7db1c4dfcaf8d13612 P 1d96bfd7d15b4150ab295c5af9efb461 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:47:05.040789 8422 raft_consensus.cc:686] T d602dc5ebae74a7db1c4dfcaf8d13612 P 1d96bfd7d15b4150ab295c5af9efb461 [term 1 LEADER]: Becoming Leader. State: Replica: 1d96bfd7d15b4150ab295c5af9efb461, State: Running, Role: LEADER
I20260619 14:47:05.041117 8422 consensus_queue.cc:237] T d602dc5ebae74a7db1c4dfcaf8d13612 P 1d96bfd7d15b4150ab295c5af9efb461 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d96bfd7d15b4150ab295c5af9efb461" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41003 } } peers { permanent_uuid: "cddbd59e8c134b38a8fb7fa3b056ad9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43399 } } peers { permanent_uuid: "273adc42fd3f43e69ef71827d279ab00" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43827 } }
I20260619 14:47:05.043663 7901 catalog_manager.cc:5495] T d602dc5ebae74a7db1c4dfcaf8d13612 P 1d96bfd7d15b4150ab295c5af9efb461 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d96bfd7d15b4150ab295c5af9efb461 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "1d96bfd7d15b4150ab295c5af9efb461" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d96bfd7d15b4150ab295c5af9efb461" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cddbd59e8c134b38a8fb7fa3b056ad9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "273adc42fd3f43e69ef71827d279ab00" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43827 } health_report { overall_health: UNKNOWN } } }
I20260619 14:47:05.056420 8459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:05.050939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60510 (local address 127.0.98.1:41003)
0619 14:47:05.051112 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:05.051120 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:05.051173 (+ 53us) server_negotiation.cc:407] Connection header received
0619 14:47:05.051277 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:05.051282 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:05.051365 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:05.051498 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:47:05.051961 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:05.052927 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:05.054216 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:05.054637 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:05.054770 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:05.055274 (+ 504us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:47:05.055947 (+ 673us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:47:05.056066 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:05.056180 (+ 114us) server_negotiation.cc:299] Negotiation successful
0619 14:47:05.056260 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 14:47:05.059893 8341 raft_consensus.cc:1261] T d602dc5ebae74a7db1c4dfcaf8d13612 P 273adc42fd3f43e69ef71827d279ab00 [term 1 FOLLOWER]: Refusing update from remote peer 1d96bfd7d15b4150ab295c5af9efb461: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:47:05.059922 8201 raft_consensus.cc:1261] T d602dc5ebae74a7db1c4dfcaf8d13612 P cddbd59e8c134b38a8fb7fa3b056ad9a [term 1 FOLLOWER]: Refusing update from remote peer 1d96bfd7d15b4150ab295c5af9efb461: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:47:05.060456 8422 consensus_queue.cc:1035] T d602dc5ebae74a7db1c4dfcaf8d13612 P 1d96bfd7d15b4150ab295c5af9efb461 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cddbd59e8c134b38a8fb7fa3b056ad9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:47:05.060752 8506 consensus_queue.cc:1035] T d602dc5ebae74a7db1c4dfcaf8d13612 P 1d96bfd7d15b4150ab295c5af9efb461 [LEADER]: Connected to new peer: Peer: permanent_uuid: "273adc42fd3f43e69ef71827d279ab00" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:47:05.073442 7904 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48958:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:47:05.073668 7904 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48958:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:47:05.076401 7904 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e3f4029a0c9b421faf47effdceacd7b5: Sending DeleteTablet for 3 replicas of tablet d602dc5ebae74a7db1c4dfcaf8d13612
I20260619 14:47:05.077224 8041 tablet_service.cc:1514] Processing DeleteTablet for tablet d602dc5ebae74a7db1c4dfcaf8d13612 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:47:05 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60508
I20260619 14:47:05.077435 8487 tablet_replica.cc:330] T d602dc5ebae74a7db1c4dfcaf8d13612 P 1d96bfd7d15b4150ab295c5af9efb461: stopping tablet replica
I20260619 14:47:05.077530 8181 tablet_service.cc:1514] Processing DeleteTablet for tablet d602dc5ebae74a7db1c4dfcaf8d13612 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:47:05 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45148
I20260619 14:47:05.077554 8487 raft_consensus.cc:2229] T d602dc5ebae74a7db1c4dfcaf8d13612 P 1d96bfd7d15b4150ab295c5af9efb461 [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:47:05.077769 8321 tablet_service.cc:1514] Processing DeleteTablet for tablet d602dc5ebae74a7db1c4dfcaf8d13612 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:47:05 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34512
I20260619 14:47:05.077983 8486 tablet_replica.cc:330] T d602dc5ebae74a7db1c4dfcaf8d13612 P cddbd59e8c134b38a8fb7fa3b056ad9a: stopping tablet replica
I20260619 14:47:05.078091 8486 raft_consensus.cc:2229] T d602dc5ebae74a7db1c4dfcaf8d13612 P cddbd59e8c134b38a8fb7fa3b056ad9a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:47:05.078205 8486 raft_consensus.cc:2258] T d602dc5ebae74a7db1c4dfcaf8d13612 P cddbd59e8c134b38a8fb7fa3b056ad9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:47:05.078094 8487 raft_consensus.cc:2258] T d602dc5ebae74a7db1c4dfcaf8d13612 P 1d96bfd7d15b4150ab295c5af9efb461 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:47:05.078433 8488 tablet_replica.cc:330] T d602dc5ebae74a7db1c4dfcaf8d13612 P 273adc42fd3f43e69ef71827d279ab00: stopping tablet replica
I20260619 14:47:05.078576 8488 raft_consensus.cc:2229] T d602dc5ebae74a7db1c4dfcaf8d13612 P 273adc42fd3f43e69ef71827d279ab00 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:47:05.078747 8488 raft_consensus.cc:2258] T d602dc5ebae74a7db1c4dfcaf8d13612 P 273adc42fd3f43e69ef71827d279ab00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:47:05.078861 7901 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48958:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:47:05.079319 7901 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:47:05.079483 8488 ts_tablet_manager.cc:1905] T d602dc5ebae74a7db1c4dfcaf8d13612 P 273adc42fd3f43e69ef71827d279ab00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:47:05.080839 8487 ts_tablet_manager.cc:1905] T d602dc5ebae74a7db1c4dfcaf8d13612 P 1d96bfd7d15b4150ab295c5af9efb461: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:47:05.082003 8488 ts_tablet_manager.cc:1918] T d602dc5ebae74a7db1c4dfcaf8d13612 P 273adc42fd3f43e69ef71827d279ab00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:47:05.082067 8488 log.cc:1196] T d602dc5ebae74a7db1c4dfcaf8d13612 P 273adc42fd3f43e69ef71827d279ab00: Deleting WAL directory at /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880383132139-392-0/minicluster-data/ts-2/wal/wals/d602dc5ebae74a7db1c4dfcaf8d13612
I20260619 14:47:05.082273 8488 ts_tablet_manager.cc:1939] T d602dc5ebae74a7db1c4dfcaf8d13612 P 273adc42fd3f43e69ef71827d279ab00: Deleting consensus metadata
I20260619 14:47:05.083464 8487 ts_tablet_manager.cc:1918] T d602dc5ebae74a7db1c4dfcaf8d13612 P 1d96bfd7d15b4150ab295c5af9efb461: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:47:05.083537 8487 log.cc:1196] T d602dc5ebae74a7db1c4dfcaf8d13612 P 1d96bfd7d15b4150ab295c5af9efb461: Deleting WAL directory at /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880383132139-392-0/minicluster-data/ts-0/wal/wals/d602dc5ebae74a7db1c4dfcaf8d13612
I20260619 14:47:05.083786 8487 ts_tablet_manager.cc:1939] T d602dc5ebae74a7db1c4dfcaf8d13612 P 1d96bfd7d15b4150ab295c5af9efb461: Deleting consensus metadata
I20260619 14:47:05.084110 7890 catalog_manager.cc:4841] TS 273adc42fd3f43e69ef71827d279ab00 (127.0.98.3:43827): tablet d602dc5ebae74a7db1c4dfcaf8d13612 (table test-table [id=d7eb752f2e564df8bf6c24c4ee95703c]) successfully deleted
I20260619 14:47:05.084628 7891 catalog_manager.cc:4841] TS 1d96bfd7d15b4150ab295c5af9efb461 (127.0.98.1:41003): tablet d602dc5ebae74a7db1c4dfcaf8d13612 (table test-table [id=d7eb752f2e564df8bf6c24c4ee95703c]) successfully deleted
I20260619 14:47:05.085019 8486 ts_tablet_manager.cc:1905] T d602dc5ebae74a7db1c4dfcaf8d13612 P cddbd59e8c134b38a8fb7fa3b056ad9a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:47:05.087847 8181 tablet_service.cc:1467] Processing CreateTablet for tablet bf59429d53af498eae0b0280911315e7 (DEFAULT_TABLE table=test-table [id=d12a74a5be7140c29d87df5d70243e87]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:47:05.087991 8041 tablet_service.cc:1467] Processing CreateTablet for tablet bf59429d53af498eae0b0280911315e7 (DEFAULT_TABLE table=test-table [id=d12a74a5be7140c29d87df5d70243e87]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:47:05.088152 8181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf59429d53af498eae0b0280911315e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:47:05.088258 8041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf59429d53af498eae0b0280911315e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:47:05.089198 8321 tablet_service.cc:1467] Processing CreateTablet for tablet bf59429d53af498eae0b0280911315e7 (DEFAULT_TABLE table=test-table [id=d12a74a5be7140c29d87df5d70243e87]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:47:05.089497 8321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf59429d53af498eae0b0280911315e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:47:05.090986 8486 ts_tablet_manager.cc:1918] T d602dc5ebae74a7db1c4dfcaf8d13612 P cddbd59e8c134b38a8fb7fa3b056ad9a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:47:05.091012 8415 tablet_bootstrap.cc:492] T bf59429d53af498eae0b0280911315e7 P 1d96bfd7d15b4150ab295c5af9efb461: Bootstrap starting.
I20260619 14:47:05.091113 8486 log.cc:1196] T d602dc5ebae74a7db1c4dfcaf8d13612 P cddbd59e8c134b38a8fb7fa3b056ad9a: Deleting WAL directory at /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880383132139-392-0/minicluster-data/ts-1/wal/wals/d602dc5ebae74a7db1c4dfcaf8d13612
I20260619 14:47:05.091403 8486 ts_tablet_manager.cc:1939] T d602dc5ebae74a7db1c4dfcaf8d13612 P cddbd59e8c134b38a8fb7fa3b056ad9a: Deleting consensus metadata
I20260619 14:47:05.092118 8415 tablet_bootstrap.cc:654] T bf59429d53af498eae0b0280911315e7 P 1d96bfd7d15b4150ab295c5af9efb461: Neither blocks nor log segments found. Creating new log.
I20260619 14:47:05.092954 7890 catalog_manager.cc:4841] TS cddbd59e8c134b38a8fb7fa3b056ad9a (127.0.98.2:43399): tablet d602dc5ebae74a7db1c4dfcaf8d13612 (table test-table [id=d7eb752f2e564df8bf6c24c4ee95703c]) successfully deleted
I20260619 14:47:05.093305 8417 tablet_bootstrap.cc:492] T bf59429d53af498eae0b0280911315e7 P 273adc42fd3f43e69ef71827d279ab00: Bootstrap starting.
I20260619 14:47:05.093849 8415 tablet_bootstrap.cc:492] T bf59429d53af498eae0b0280911315e7 P 1d96bfd7d15b4150ab295c5af9efb461: No bootstrap required, opened a new log
I20260619 14:47:05.093976 8415 ts_tablet_manager.cc:1397] T bf59429d53af498eae0b0280911315e7 P 1d96bfd7d15b4150ab295c5af9efb461: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 14:47:05.094174 8417 tablet_bootstrap.cc:654] T bf59429d53af498eae0b0280911315e7 P 273adc42fd3f43e69ef71827d279ab00: Neither blocks nor log segments found. Creating new log.
I20260619 14:47:05.094487 8415 raft_consensus.cc:348] T bf59429d53af498eae0b0280911315e7 P 1d96bfd7d15b4150ab295c5af9efb461 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d96bfd7d15b4150ab295c5af9efb461" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41003 } } peers { permanent_uuid: "cddbd59e8c134b38a8fb7fa3b056ad9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43399 } } peers { permanent_uuid: "273adc42fd3f43e69ef71827d279ab00" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43827 } }
I20260619 14:47:05.094674 8415 raft_consensus.cc:374] T bf59429d53af498eae0b0280911315e7 P 1d96bfd7d15b4150ab295c5af9efb461 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:47:05.094728 8415 raft_consensus.cc:729] T bf59429d53af498eae0b0280911315e7 P 1d96bfd7d15b4150ab295c5af9efb461 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d96bfd7d15b4150ab295c5af9efb461, State: Initialized, Role: FOLLOWER
I20260619 14:47:05.094894 8415 consensus_queue.cc:260] T bf59429d53af498eae0b0280911315e7 P 1d96bfd7d15b4150ab295c5af9efb461 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d96bfd7d15b4150ab295c5af9efb461" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41003 } } peers { permanent_uuid: "cddbd59e8c134b38a8fb7fa3b056ad9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43399 } } peers { permanent_uuid: "273adc42fd3f43e69ef71827d279ab00" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43827 } }
I20260619 14:47:05.095227 8415 ts_tablet_manager.cc:1428] T bf59429d53af498eae0b0280911315e7 P 1d96bfd7d15b4150ab295c5af9efb461: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260619 14:47:05.095456 8417 tablet_bootstrap.cc:492] T bf59429d53af498eae0b0280911315e7 P 273adc42fd3f43e69ef71827d279ab00: No bootstrap required, opened a new log
I20260619 14:47:05.095541 8417 ts_tablet_manager.cc:1397] T bf59429d53af498eae0b0280911315e7 P 273adc42fd3f43e69ef71827d279ab00: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260619 14:47:05.095624 8416 tablet_bootstrap.cc:492] T bf59429d53af498eae0b0280911315e7 P cddbd59e8c134b38a8fb7fa3b056ad9a: Bootstrap starting.
I20260619 14:47:05.095906 8417 raft_consensus.cc:348] T bf59429d53af498eae0b0280911315e7 P 273adc42fd3f43e69ef71827d279ab00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d96bfd7d15b4150ab295c5af9efb461" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41003 } } peers { permanent_uuid: "cddbd59e8c134b38a8fb7fa3b056ad9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43399 } } peers { permanent_uuid: "273adc42fd3f43e69ef71827d279ab00" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43827 } }
I20260619 14:47:05.096015 8417 raft_consensus.cc:374] T bf59429d53af498eae0b0280911315e7 P 273adc42fd3f43e69ef71827d279ab00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:47:05.096050 8417 raft_consensus.cc:729] T bf59429d53af498eae0b0280911315e7 P 273adc42fd3f43e69ef71827d279ab00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 273adc42fd3f43e69ef71827d279ab00, State: Initialized, Role: FOLLOWER
I20260619 14:47:05.096206 8417 consensus_queue.cc:260] T bf59429d53af498eae0b0280911315e7 P 273adc42fd3f43e69ef71827d279ab00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d96bfd7d15b4150ab295c5af9efb461" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41003 } } peers { permanent_uuid: "cddbd59e8c134b38a8fb7fa3b056ad9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43399 } } peers { permanent_uuid: "273adc42fd3f43e69ef71827d279ab00" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43827 } }
I20260619 14:47:05.096479 8417 ts_tablet_manager.cc:1428] T bf59429d53af498eae0b0280911315e7 P 273adc42fd3f43e69ef71827d279ab00: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:47:05.096876 8416 tablet_bootstrap.cc:654] T bf59429d53af498eae0b0280911315e7 P cddbd59e8c134b38a8fb7fa3b056ad9a: Neither blocks nor log segments found. Creating new log.
I20260619 14:47:05.098448 8416 tablet_bootstrap.cc:492] T bf59429d53af498eae0b0280911315e7 P cddbd59e8c134b38a8fb7fa3b056ad9a: No bootstrap required, opened a new log
I20260619 14:47:05.098572 8416 ts_tablet_manager.cc:1397] T bf59429d53af498eae0b0280911315e7 P cddbd59e8c134b38a8fb7fa3b056ad9a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 14:47:05.099031 8416 raft_consensus.cc:348] T bf59429d53af498eae0b0280911315e7 P cddbd59e8c134b38a8fb7fa3b056ad9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d96bfd7d15b4150ab295c5af9efb461" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41003 } } peers { permanent_uuid: "cddbd59e8c134b38a8fb7fa3b056ad9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43399 } } peers { permanent_uuid: "273adc42fd3f43e69ef71827d279ab00" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43827 } }
I20260619 14:47:05.099169 8416 raft_consensus.cc:374] T bf59429d53af498eae0b0280911315e7 P cddbd59e8c134b38a8fb7fa3b056ad9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:47:05.099212 8416 raft_consensus.cc:729] T bf59429d53af498eae0b0280911315e7 P cddbd59e8c134b38a8fb7fa3b056ad9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cddbd59e8c134b38a8fb7fa3b056ad9a, State: Initialized, Role: FOLLOWER
I20260619 14:47:05.099387 8416 consensus_queue.cc:260] T bf59429d53af498eae0b0280911315e7 P cddbd59e8c134b38a8fb7fa3b056ad9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d96bfd7d15b4150ab295c5af9efb461" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41003 } } peers { permanent_uuid: "cddbd59e8c134b38a8fb7fa3b056ad9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43399 } } peers { permanent_uuid: "273adc42fd3f43e69ef71827d279ab00" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43827 } }
I20260619 14:47:05.100737 8416 ts_tablet_manager.cc:1428] T bf59429d53af498eae0b0280911315e7 P cddbd59e8c134b38a8fb7fa3b056ad9a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260619 14:47:05.260397 8473 raft_consensus.cc:482] T bf59429d53af498eae0b0280911315e7 P 273adc42fd3f43e69ef71827d279ab00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:47:05.260543 8473 raft_consensus.cc:504] T bf59429d53af498eae0b0280911315e7 P 273adc42fd3f43e69ef71827d279ab00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d96bfd7d15b4150ab295c5af9efb461" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41003 } } peers { permanent_uuid: "cddbd59e8c134b38a8fb7fa3b056ad9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43399 } } peers { permanent_uuid: "273adc42fd3f43e69ef71827d279ab00" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43827 } }
I20260619 14:47:05.261109 8473 leader_election.cc:288] T bf59429d53af498eae0b0280911315e7 P 273adc42fd3f43e69ef71827d279ab00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d96bfd7d15b4150ab295c5af9efb461 (127.0.98.1:41003), cddbd59e8c134b38a8fb7fa3b056ad9a (127.0.98.2:43399)
I20260619 14:47:05.261700 8061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf59429d53af498eae0b0280911315e7" candidate_uuid: "273adc42fd3f43e69ef71827d279ab00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d96bfd7d15b4150ab295c5af9efb461" is_pre_election: true
I20260619 14:47:05.261749 8201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf59429d53af498eae0b0280911315e7" candidate_uuid: "273adc42fd3f43e69ef71827d279ab00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cddbd59e8c134b38a8fb7fa3b056ad9a" is_pre_election: true
I20260619 14:47:05.261896 8061 raft_consensus.cc:2454] T bf59429d53af498eae0b0280911315e7 P 1d96bfd7d15b4150ab295c5af9efb461 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 273adc42fd3f43e69ef71827d279ab00 in term 0.
I20260619 14:47:05.261901 8201 raft_consensus.cc:2454] T bf59429d53af498eae0b0280911315e7 P cddbd59e8c134b38a8fb7fa3b056ad9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 273adc42fd3f43e69ef71827d279ab00 in term 0.
I20260619 14:47:05.262300 8275 leader_election.cc:302] T bf59429d53af498eae0b0280911315e7 P 273adc42fd3f43e69ef71827d279ab00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d96bfd7d15b4150ab295c5af9efb461, 273adc42fd3f43e69ef71827d279ab00; no voters:
I20260619 14:47:05.262488 8473 raft_consensus.cc:2781] T bf59429d53af498eae0b0280911315e7 P 273adc42fd3f43e69ef71827d279ab00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:47:05.262552 8473 raft_consensus.cc:482] T bf59429d53af498eae0b0280911315e7 P 273adc42fd3f43e69ef71827d279ab00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:47:05.262591 8473 raft_consensus.cc:3037] T bf59429d53af498eae0b0280911315e7 P 273adc42fd3f43e69ef71827d279ab00 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:47:05.263842 8473 raft_consensus.cc:504] T bf59429d53af498eae0b0280911315e7 P 273adc42fd3f43e69ef71827d279ab00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d96bfd7d15b4150ab295c5af9efb461" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41003 } } peers { permanent_uuid: "cddbd59e8c134b38a8fb7fa3b056ad9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43399 } } peers { permanent_uuid: "273adc42fd3f43e69ef71827d279ab00" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43827 } }
I20260619 14:47:05.264233 8473 leader_election.cc:288] T bf59429d53af498eae0b0280911315e7 P 273adc42fd3f43e69ef71827d279ab00 [CANDIDATE]: Term 1 election: Requested vote from peers 1d96bfd7d15b4150ab295c5af9efb461 (127.0.98.1:41003), cddbd59e8c134b38a8fb7fa3b056ad9a (127.0.98.2:43399)
I20260619 14:47:05.264705 8061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf59429d53af498eae0b0280911315e7" candidate_uuid: "273adc42fd3f43e69ef71827d279ab00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d96bfd7d15b4150ab295c5af9efb461"
I20260619 14:47:05.264837 8061 raft_consensus.cc:3037] T bf59429d53af498eae0b0280911315e7 P 1d96bfd7d15b4150ab295c5af9efb461 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:47:05.264986 8201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf59429d53af498eae0b0280911315e7" candidate_uuid: "273adc42fd3f43e69ef71827d279ab00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cddbd59e8c134b38a8fb7fa3b056ad9a"
I20260619 14:47:05.265127 8201 raft_consensus.cc:3037] T bf59429d53af498eae0b0280911315e7 P cddbd59e8c134b38a8fb7fa3b056ad9a [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:47:05.266095 8061 raft_consensus.cc:2454] T bf59429d53af498eae0b0280911315e7 P 1d96bfd7d15b4150ab295c5af9efb461 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 273adc42fd3f43e69ef71827d279ab00 in term 1.
I20260619 14:47:05.266296 8201 raft_consensus.cc:2454] T bf59429d53af498eae0b0280911315e7 P cddbd59e8c134b38a8fb7fa3b056ad9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 273adc42fd3f43e69ef71827d279ab00 in term 1.
I20260619 14:47:05.266517 8275 leader_election.cc:302] T bf59429d53af498eae0b0280911315e7 P 273adc42fd3f43e69ef71827d279ab00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d96bfd7d15b4150ab295c5af9efb461, 273adc42fd3f43e69ef71827d279ab00; no voters:
I20260619 14:47:05.266702 8473 raft_consensus.cc:2781] T bf59429d53af498eae0b0280911315e7 P 273adc42fd3f43e69ef71827d279ab00 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:47:05.266887 8473 raft_consensus.cc:686] T bf59429d53af498eae0b0280911315e7 P 273adc42fd3f43e69ef71827d279ab00 [term 1 LEADER]: Becoming Leader. State: Replica: 273adc42fd3f43e69ef71827d279ab00, State: Running, Role: LEADER
I20260619 14:47:05.267036 8473 consensus_queue.cc:237] T bf59429d53af498eae0b0280911315e7 P 273adc42fd3f43e69ef71827d279ab00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d96bfd7d15b4150ab295c5af9efb461" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41003 } } peers { permanent_uuid: "cddbd59e8c134b38a8fb7fa3b056ad9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43399 } } peers { permanent_uuid: "273adc42fd3f43e69ef71827d279ab00" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43827 } }
I20260619 14:47:05.268973 7904 catalog_manager.cc:5495] T bf59429d53af498eae0b0280911315e7 P 273adc42fd3f43e69ef71827d279ab00 reported cstate change: term changed from 0 to 1, leader changed from <none> to 273adc42fd3f43e69ef71827d279ab00 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "273adc42fd3f43e69ef71827d279ab00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d96bfd7d15b4150ab295c5af9efb461" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cddbd59e8c134b38a8fb7fa3b056ad9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "273adc42fd3f43e69ef71827d279ab00" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43827 } health_report { overall_health: HEALTHY } } }
I20260619 14:47:05.300112 8465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:05.296859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34522 (local address 127.0.98.3:43827)
0619 14:47:05.296982 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:05.296986 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:05.296998 (+ 12us) server_negotiation.cc:407] Connection header received
0619 14:47:05.297086 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:05.297089 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:05.297134 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:05.297204 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:47:05.297638 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:05.298235 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:05.299223 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:05.299433 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:05.299516 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:05.299741 (+ 225us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:47:05.299811 (+ 70us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:47:05.299865 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:05.299950 (+ 85us) server_negotiation.cc:299] Negotiation successful
0619 14:47:05.299998 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 14:47:05.310441 8465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:05.307170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48059 (local address 127.0.98.3:43827)
0619 14:47:05.307308 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:05.307313 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:05.307408 (+ 95us) server_negotiation.cc:407] Connection header received
0619 14:47:05.307660 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:05.307663 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:05.307712 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:05.307792 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:47:05.308078 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:05.308582 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:05.309413 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:05.309626 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:05.309735 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:05.309954 (+ 219us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:47:05.310046 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:47:05.310124 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:05.310210 (+ 86us) server_negotiation.cc:299] Negotiation successful
0619 14:47:05.310287 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":68}
I20260619 14:47:05.310441 8483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:05.307083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43827 (local address 127.0.98.3:48059)
0619 14:47:05.307312 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:05.307335 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:05.307549 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:05.307798 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:05.307800 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:05.307806 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0619 14:47:05.307960 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:05.307966 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:05.308706 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:05.308712 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:05.309293 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:05.309301 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:05.309734 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:05.309737 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:05.309812 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:05.309836 (+ 24us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0619 14:47:05.310135 (+ 299us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0619 14:47:05.310143 (+ 8us) client_negotiation.cc:769] Sending connection context
0619 14:47:05.310204 (+ 61us) client_negotiation.cc:240] Negotiation successful
0619 14:47:05.310321 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65,"mutex_wait_us":44}
I20260619 14:47:05.317703 8061 raft_consensus.cc:1261] T bf59429d53af498eae0b0280911315e7 P 1d96bfd7d15b4150ab295c5af9efb461 [term 1 FOLLOWER]: Refusing update from remote peer 273adc42fd3f43e69ef71827d279ab00: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:47:05.317870 8201 raft_consensus.cc:1261] T bf59429d53af498eae0b0280911315e7 P cddbd59e8c134b38a8fb7fa3b056ad9a [term 1 FOLLOWER]: Refusing update from remote peer 273adc42fd3f43e69ef71827d279ab00: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:47:05.318207 8460 consensus_queue.cc:1035] T bf59429d53af498eae0b0280911315e7 P 273adc42fd3f43e69ef71827d279ab00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d96bfd7d15b4150ab295c5af9efb461" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:47:05.318562 8460 consensus_queue.cc:1035] T bf59429d53af498eae0b0280911315e7 P 273adc42fd3f43e69ef71827d279ab00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cddbd59e8c134b38a8fb7fa3b056ad9a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:47:05.359400 7904 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48958:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:47:05.359588 7904 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48958:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:47:05.362105 7904 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e3f4029a0c9b421faf47effdceacd7b5: Sending DeleteTablet for 3 replicas of tablet bf59429d53af498eae0b0280911315e7
I20260619 14:47:05.362736 8041 tablet_service.cc:1514] Processing DeleteTablet for tablet bf59429d53af498eae0b0280911315e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:47:05 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60508
I20260619 14:47:05.362802 8321 tablet_service.cc:1514] Processing DeleteTablet for tablet bf59429d53af498eae0b0280911315e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:47:05 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34512
I20260619 14:47:05.362934 8487 tablet_replica.cc:330] T bf59429d53af498eae0b0280911315e7 P 1d96bfd7d15b4150ab295c5af9efb461: stopping tablet replica
I20260619 14:47:05.363003 8488 tablet_replica.cc:330] T bf59429d53af498eae0b0280911315e7 P 273adc42fd3f43e69ef71827d279ab00: stopping tablet replica
I20260619 14:47:05.363044 8487 raft_consensus.cc:2229] T bf59429d53af498eae0b0280911315e7 P 1d96bfd7d15b4150ab295c5af9efb461 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:47:05.363119 8488 raft_consensus.cc:2229] T bf59429d53af498eae0b0280911315e7 P 273adc42fd3f43e69ef71827d279ab00 [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:47:05.363161 8487 raft_consensus.cc:2258] T bf59429d53af498eae0b0280911315e7 P 1d96bfd7d15b4150ab295c5af9efb461 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:47:05.363193 8181 tablet_service.cc:1514] Processing DeleteTablet for tablet bf59429d53af498eae0b0280911315e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:47:05 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45148
I20260619 14:47:05.363457 8486 tablet_replica.cc:330] T bf59429d53af498eae0b0280911315e7 P cddbd59e8c134b38a8fb7fa3b056ad9a: stopping tablet replica
I20260619 14:47:05.363493 8488 raft_consensus.cc:2258] T bf59429d53af498eae0b0280911315e7 P 273adc42fd3f43e69ef71827d279ab00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:47:05.363585 8486 raft_consensus.cc:2229] T bf59429d53af498eae0b0280911315e7 P cddbd59e8c134b38a8fb7fa3b056ad9a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:47:05.363704 8486 raft_consensus.cc:2258] T bf59429d53af498eae0b0280911315e7 P cddbd59e8c134b38a8fb7fa3b056ad9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:47:05.364315 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 7976
I20260619 14:47:05.364447 8488 ts_tablet_manager.cc:1905] T bf59429d53af498eae0b0280911315e7 P 273adc42fd3f43e69ef71827d279ab00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:47:05.364567 8487 ts_tablet_manager.cc:1905] T bf59429d53af498eae0b0280911315e7 P 1d96bfd7d15b4150ab295c5af9efb461: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:47:05.366892 8488 ts_tablet_manager.cc:1918] T bf59429d53af498eae0b0280911315e7 P 273adc42fd3f43e69ef71827d279ab00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260619 14:47:05.366957 8488 log.cc:1196] T bf59429d53af498eae0b0280911315e7 P 273adc42fd3f43e69ef71827d279ab00: Deleting WAL directory at /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781880383132139-392-0/minicluster-data/ts-2/wal/wals/bf59429d53af498eae0b0280911315e7
I20260619 14:47:05.367158 8488 ts_tablet_manager.cc:1939] T bf59429d53af498eae0b0280911315e7 P 273adc42fd3f43e69ef71827d279ab00: Deleting consensus metadata
I20260619 14:47:05.368512 7890 catalog_manager.cc:4841] TS 273adc42fd3f43e69ef71827d279ab00 (127.0.98.3:43827): tablet bf59429d53af498eae0b0280911315e7 (table test-table [id=d12a74a5be7140c29d87df5d70243e87]) successfully deleted
I20260619 14:47:05.373307 8486 ts_tablet_manager.cc:1905] T bf59429d53af498eae0b0280911315e7 P cddbd59e8c134b38a8fb7fa3b056ad9a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260619 14:47:05.374802 7891 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41003 (error 108)
I20260619 14:47:05.375990 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 8116
I20260619 14:47:05.377038 8522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:05.376040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41003 (local address 127.0.0.1:60518)
0619 14:47:05.376810 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:05.376898 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":671,"thread_start_us":64,"threads_started":1}
W20260619 14:47:05.377388 7891 catalog_manager.cc:4584] TS 1d96bfd7d15b4150ab295c5af9efb461 (127.0.98.1:41003): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bf59429d53af498eae0b0280911315e7: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41003: connect: Connection refused (error 111)
I20260619 14:47:05.384377 8522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:05.384061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43399 (local address 127.0.0.1:45164)
0619 14:47:05.384197 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:05.384261 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:43399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260619 14:47:05.384536 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 8256
I20260619 14:47:05.392064 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 7872
2026-06-19T14:47:05Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3710 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781880383132139-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:47:05 dist-test-slave-2sb1 krb5kdc[8525](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:47:05 dist-test-slave-2sb1 krb5kdc[8525](info): set up 2 sockets
Jun 19 14:47:05 dist-test-slave-2sb1 krb5kdc[8525](info): commencing operation
krb5kdc: starting...
W20260619 14:47:06.510221 392 mini_kdc.cc:121] Time spent starting KDC: real 1.094s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781880383132139-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781880383132139-392-0/krb5kdc/test-user.keytab.
Jun 19 14:47:06 dist-test-slave-2sb1 krb5kdc[8525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880426, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:47:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:47:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 14:47:06.654387 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781880383132139-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781880383132139-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781880383132139-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32843
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32843
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:47:06.768903 8538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:47:06.769205 8538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:47:06.769248 8538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:47:06.773469 8538 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:47:06.773547 8538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:47:06.773567 8538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:47:06.773584 8538 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:47:06.773600 8538 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:47:06.779300 8538 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781880383132139-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781880383132139-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32843
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32843
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8538
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781880383132139-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:47:06.780517 8538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:47:06.781659 8538 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:47:06.790992 8546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:47:06.790939 8543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:47:06.790956 8544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:47:06.791435 8538 server_base.cc:1034] running on GCE node
I20260619 14:47:06.792022 8538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:47:06.793439 8538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:47:06.794651 8538 hybrid_clock.cc:648] HybridClock initialized: now 1781880426794618 us; error 48 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-19T14:47:06Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1409 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:47:06 dist-test-slave-2sb1 krb5kdc[8551](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:47:06 dist-test-slave-2sb1 krb5kdc[8551](info): set up 2 sockets
Jun 19 14:47:06 dist-test-slave-2sb1 krb5kdc[8551](info): commencing operation
krb5kdc: starting...
W20260619 14:47:07.914186 392 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/krb5kdc/test-user.keytab.
Jun 19 14:47:08 dist-test-slave-2sb1 krb5kdc[8551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880428, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:47:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:47:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 14:47:08.057319 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43861
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43861
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:47:08.183404 8564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:47:08.183796 8564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:47:08.183878 8564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:47:08.188467 8564 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:47:08.188602 8564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:47:08.188639 8564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:47:08.188710 8564 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:47:08.188761 8564 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:47:08.194538 8564 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43861
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8564
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:47:08.195798 8564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:47:08.196882 8564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:47:08.206079 8569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:47:08.206137 8570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:47:08.206167 8572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:47:08.206331 8564 server_base.cc:1034] running on GCE node
I20260619 14:47:08.207407 8564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:47:08.208818 8564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:47:08.210047 8564 hybrid_clock.cc:648] HybridClock initialized: now 1781880428210027 us; error 42 us; skew 500 ppm
Jun 19 14:47:08 dist-test-slave-2sb1 krb5kdc[8551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880428, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:47:08.213505 8564 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 14:47:08.214653 8564 webserver.cc:460] Webserver started at http://127.0.98.62:45907/ using document root <none> and password file <none>
I20260619 14:47:08.215270 8564 fs_manager.cc:359] Metadata directory not provided
I20260619 14:47:08.215369 8564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:47:08.215612 8564 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:47:08.217912 8564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/master-0/data/instance:
uuid: "457c5d49e6e9463199d5eb8401fae9db"
format_stamp: "Formatted at 2026-06-19 14:47:08 on dist-test-slave-2sb1"
server_key: "d87066ea5b37bc0006a0dd0cceb559a2"
server_key_iv: "2909e8c47769e2300dba18252402a44d"
server_key_version: "encryptionkey@0"
I20260619 14:47:08.218542 8564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/master-0/wal/instance:
uuid: "457c5d49e6e9463199d5eb8401fae9db"
format_stamp: "Formatted at 2026-06-19 14:47:08 on dist-test-slave-2sb1"
server_key: "d87066ea5b37bc0006a0dd0cceb559a2"
server_key_iv: "2909e8c47769e2300dba18252402a44d"
server_key_version: "encryptionkey@0"
I20260619 14:47:08.222738 8564 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260619 14:47:08.226203 8579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:47:08.227957 8564 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 14:47:08.228137 8564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/master-0/wal
uuid: "457c5d49e6e9463199d5eb8401fae9db"
format_stamp: "Formatted at 2026-06-19 14:47:08 on dist-test-slave-2sb1"
server_key: "d87066ea5b37bc0006a0dd0cceb559a2"
server_key_iv: "2909e8c47769e2300dba18252402a44d"
server_key_version: "encryptionkey@0"
I20260619 14:47:08.228278 8564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:47:08.254386 8564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:47:08.257925 8564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:47:08.258255 8564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:47:08.265735 8564 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43861
I20260619 14:47:08.265803 8631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43861 every 8 connection(s)
I20260619 14:47:08.266759 8564 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:47:08.269804 8632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:47:08.276132 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 8564
I20260619 14:47:08.276284 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/master-0/wal/instance
I20260619 14:47:08.276407 8632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 457c5d49e6e9463199d5eb8401fae9db: Bootstrap starting.
I20260619 14:47:08.276679 392 external_mini_cluster.cc:1428] Setting key f25a4cc0711d962a2c8af726e49f7388
I20260619 14:47:08.279567 8632 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 457c5d49e6e9463199d5eb8401fae9db: Neither blocks nor log segments found. Creating new log.
I20260619 14:47:08.280519 8632 log.cc:824] T 00000000000000000000000000000000 P 457c5d49e6e9463199d5eb8401fae9db: Log is configured to *not* fsync() on all Append() calls
I20260619 14:47:08.282806 8632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 457c5d49e6e9463199d5eb8401fae9db: No bootstrap required, opened a new log
I20260619 14:47:08.287794 8632 raft_consensus.cc:348] T 00000000000000000000000000000000 P 457c5d49e6e9463199d5eb8401fae9db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "457c5d49e6e9463199d5eb8401fae9db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43861 } }
I20260619 14:47:08.288355 8632 raft_consensus.cc:374] T 00000000000000000000000000000000 P 457c5d49e6e9463199d5eb8401fae9db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:47:08.288460 8632 raft_consensus.cc:729] T 00000000000000000000000000000000 P 457c5d49e6e9463199d5eb8401fae9db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 457c5d49e6e9463199d5eb8401fae9db, State: Initialized, Role: FOLLOWER
Jun 19 14:47:08 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880428, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:47:08.289191 8632 consensus_queue.cc:260] T 00000000000000000000000000000000 P 457c5d49e6e9463199d5eb8401fae9db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "457c5d49e6e9463199d5eb8401fae9db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43861 } }
I20260619 14:47:08.289367 8632 raft_consensus.cc:388] T 00000000000000000000000000000000 P 457c5d49e6e9463199d5eb8401fae9db [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:47:08.289438 8632 raft_consensus.cc:482] T 00000000000000000000000000000000 P 457c5d49e6e9463199d5eb8401fae9db [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:47:08.289546 8632 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 457c5d49e6e9463199d5eb8401fae9db [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:47:08.290818 8632 raft_consensus.cc:504] T 00000000000000000000000000000000 P 457c5d49e6e9463199d5eb8401fae9db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "457c5d49e6e9463199d5eb8401fae9db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43861 } }
I20260619 14:47:08.291211 8632 leader_election.cc:302] T 00000000000000000000000000000000 P 457c5d49e6e9463199d5eb8401fae9db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 457c5d49e6e9463199d5eb8401fae9db; no voters:
I20260619 14:47:08.291594 8632 leader_election.cc:288] T 00000000000000000000000000000000 P 457c5d49e6e9463199d5eb8401fae9db [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:47:08.291703 8637 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 457c5d49e6e9463199d5eb8401fae9db [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:47:08.291947 8637 raft_consensus.cc:686] T 00000000000000000000000000000000 P 457c5d49e6e9463199d5eb8401fae9db [term 1 LEADER]: Becoming Leader. State: Replica: 457c5d49e6e9463199d5eb8401fae9db, State: Running, Role: LEADER
I20260619 14:47:08.292382 8637 consensus_queue.cc:237] T 00000000000000000000000000000000 P 457c5d49e6e9463199d5eb8401fae9db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "457c5d49e6e9463199d5eb8401fae9db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43861 } }
I20260619 14:47:08.293154 8632 sys_catalog.cc:564] T 00000000000000000000000000000000 P 457c5d49e6e9463199d5eb8401fae9db [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:47:08.294529 8639 sys_catalog.cc:455] T 00000000000000000000000000000000 P 457c5d49e6e9463199d5eb8401fae9db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "457c5d49e6e9463199d5eb8401fae9db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "457c5d49e6e9463199d5eb8401fae9db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43861 } } }
I20260619 14:47:08.294932 8639 sys_catalog.cc:458] T 00000000000000000000000000000000 P 457c5d49e6e9463199d5eb8401fae9db [sys.catalog]: This master's current role is: LEADER
I20260619 14:47:08.294834 8638 sys_catalog.cc:455] T 00000000000000000000000000000000 P 457c5d49e6e9463199d5eb8401fae9db [sys.catalog]: SysCatalogTable state changed. Reason: New leader 457c5d49e6e9463199d5eb8401fae9db. Latest consensus state: current_term: 1 leader_uuid: "457c5d49e6e9463199d5eb8401fae9db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "457c5d49e6e9463199d5eb8401fae9db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43861 } } }
I20260619 14:47:08.295258 8638 sys_catalog.cc:458] T 00000000000000000000000000000000 P 457c5d49e6e9463199d5eb8401fae9db [sys.catalog]: This master's current role is: LEADER
I20260619 14:47:08.295868 8635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:08.278369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49214 (local address 127.0.98.62:43861)
0619 14:47:08.279124 (+ 755us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:08.279152 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:08.279209 (+ 57us) server_negotiation.cc:407] Connection header received
0619 14:47:08.280211 (+ 1002us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:08.280252 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:08.281010 (+ 758us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:08.281460 (+ 450us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:08.282383 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:08.283481 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:08.284461 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:08.285082 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:08.285280 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:08.289348 (+ 4068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:47:08.289356 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:47:08.289376 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:47:08.289464 (+ 88us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:47:08.291724 (+ 2260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:08.292359 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:08.292365 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:08.292372 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:08.292554 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:08.292946 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:08.292949 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:08.292951 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:08.293470 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:47:08.293697 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:08.294363 (+ 666us) server_negotiation.cc:299] Negotiation successful
0619 14:47:08.294689 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":201,"threads_started":1}
W20260619 14:47:08.298559 8652 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 457c5d49e6e9463199d5eb8401fae9db: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:47:08.298659 8652 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:47:08.298746 8651 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:47:08.299497 8651 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:47:08.305855 8651 catalog_manager.cc:1347] Generated new cluster ID: e58eca3524b9443e91ba5804a523970a
I20260619 14:47:08.306038 8651 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:47:08.314237 8651 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:47:08.315335 8651 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:47:08.326922 8651 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 457c5d49e6e9463199d5eb8401fae9db: Generated new TSK 0
I20260619 14:47:08.328042 8651 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 14:47:08.373677 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43861
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:47:08.496945 8658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:47:08.497268 8658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:47:08.497320 8658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:47:08.501905 8658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:47:08.501999 8658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:47:08.502065 8658 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 14:47:08.507668 8658 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43861
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8658
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:47:08.508787 8658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:47:08.509917 8658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:47:08.520987 8666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:47:08.521095 8664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:47:08.521095 8663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:47:08.521394 8658 server_base.cc:1034] running on GCE node
I20260619 14:47:08.522032 8658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:47:08.522784 8658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:47:08.523989 8658 hybrid_clock.cc:648] HybridClock initialized: now 1781880428523961 us; error 50 us; skew 500 ppm
Jun 19 14:47:08 dist-test-slave-2sb1 krb5kdc[8551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880428, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:47:08.527354 8658 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 14:47:08.528363 8658 webserver.cc:460] Webserver started at http://127.0.98.1:41491/ using document root <none> and password file <none>
I20260619 14:47:08.529002 8658 fs_manager.cc:359] Metadata directory not provided
I20260619 14:47:08.529076 8658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:47:08.529260 8658 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:47:08.531472 8658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/ts-0/data/instance:
uuid: "e7c951ef1c0b4804a75f903a9d22a229"
format_stamp: "Formatted at 2026-06-19 14:47:08 on dist-test-slave-2sb1"
server_key: "4b7cd6af151488754bb5614d0a01f96a"
server_key_iv: "18774166599f14831c17fc316d6b0b09"
server_key_version: "encryptionkey@0"
I20260619 14:47:08.532169 8658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e7c951ef1c0b4804a75f903a9d22a229"
format_stamp: "Formatted at 2026-06-19 14:47:08 on dist-test-slave-2sb1"
server_key: "4b7cd6af151488754bb5614d0a01f96a"
server_key_iv: "18774166599f14831c17fc316d6b0b09"
server_key_version: "encryptionkey@0"
I20260619 14:47:08.536526 8658 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 14:47:08.539587 8673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:47:08.541281 8658 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260619 14:47:08.541430 8658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/ts-0/wal
uuid: "e7c951ef1c0b4804a75f903a9d22a229"
format_stamp: "Formatted at 2026-06-19 14:47:08 on dist-test-slave-2sb1"
server_key: "4b7cd6af151488754bb5614d0a01f96a"
server_key_iv: "18774166599f14831c17fc316d6b0b09"
server_key_version: "encryptionkey@0"
I20260619 14:47:08.541574 8658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:47:08.563225 8658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:47:08.567044 8658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:47:08.567268 8658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:47:08.567920 8658 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:47:08.568876 8658 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:47:08.568926 8658 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:47:08.568994 8658 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:47:08.569027 8658 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:47:08.579562 8658 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46849
I20260619 14:47:08.579594 8786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46849 every 8 connection(s)
I20260619 14:47:08.580538 8658 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/ts-0/data/info.pb
I20260619 14:47:08.581418 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 8658
I20260619 14:47:08.581586 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/ts-0/wal/instance
I20260619 14:47:08.581931 392 external_mini_cluster.cc:1428] Setting key 6156fc853f3ea25f619f4b67202bd340
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Jun 19 14:47:08 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880428, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 14:47:08.597827 8635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:08.582744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51617 (local address 127.0.98.62:43861)
0619 14:47:08.582889 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:08.582893 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:08.583893 (+ 1000us) server_negotiation.cc:407] Connection header received
0619 14:47:08.585440 (+ 1547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:08.585445 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:08.585524 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:08.585638 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:08.587086 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:08.587686 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:08.588965 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:08.589217 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:08.589338 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:08.592282 (+ 2944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:47:08.592285 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:47:08.592287 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:47:08.592314 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:47:08.594170 (+ 1856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:08.595154 (+ 984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:08.595159 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:08.595164 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:08.595291 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:08.595868 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:08.595876 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:08.595881 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:08.596301 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:47:08.596478 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:08.597503 (+ 1025us) server_negotiation.cc:299] Negotiation successful
0619 14:47:08.597649 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 14:47:08.598832 8789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:08.583197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43861 (local address 127.0.98.1:51617)
0619 14:47:08.583705 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:08.583764 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:08.585199 (+ 1435us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:08.585959 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:08.585967 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:08.586405 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:47:08.586918 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:08.586931 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:08.587829 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:08.587832 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:08.588832 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:08.588842 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:08.589386 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:08.589390 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:08.589578 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:08.590326 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:47:08.590344 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:47:08.592077 (+ 1733us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:47:08.594437 (+ 2360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:47:08.594446 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:47:08.594474 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:47:08.594942 (+ 468us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:47:08.595425 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:47:08.595429 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:47:08.595431 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:47:08.595623 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:47:08.596517 (+ 894us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:47:08.596523 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:47:08.597214 (+ 691us) client_negotiation.cc:769] Sending connection context
0619 14:47:08.597445 (+ 231us) client_negotiation.cc:240] Negotiation successful
0619 14:47:08.597817 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":167,"threads_started":1}
I20260619 14:47:08.600507 8787 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43861
I20260619 14:47:08.600864 8787 heartbeater.cc:464] Registering TS with master...
I20260619 14:47:08.601415 8787 heartbeater.cc:511] Master 127.0.98.62:43861 requested a full tablet report, sending...
I20260619 14:47:08.603200 8595 ts_manager.cc:194] Registered new tserver with Master: e7c951ef1c0b4804a75f903a9d22a229 (127.0.98.1:46849)
I20260619 14:47:08.604892 8595 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:51617
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 14:47:08.615805 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43861
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:47:08.736804 8792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:47:08.737142 8792 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:47:08.737218 8792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:47:08.741554 8792 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:47:08.741675 8792 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:47:08.741776 8792 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 14:47:08.747646 8792 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43861
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8792
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:47:08.748807 8792 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:47:08.749897 8792 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:47:08.760349 8800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:47:08.760437 8797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:47:08.760349 8798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:47:08.760756 8792 server_base.cc:1034] running on GCE node
I20260619 14:47:08.761233 8792 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:47:08.761993 8792 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:47:08.763211 8792 hybrid_clock.cc:648] HybridClock initialized: now 1781880428763174 us; error 55 us; skew 500 ppm
Jun 19 14:47:08 dist-test-slave-2sb1 krb5kdc[8551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880428, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:47:08.766928 8792 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 14:47:08.767970 8792 webserver.cc:460] Webserver started at http://127.0.98.2:40701/ using document root <none> and password file <none>
I20260619 14:47:08.768612 8792 fs_manager.cc:359] Metadata directory not provided
I20260619 14:47:08.768754 8792 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:47:08.769002 8792 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:47:08.771153 8792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/ts-1/data/instance:
uuid: "d0938919e2a248739b6179b3305cd623"
format_stamp: "Formatted at 2026-06-19 14:47:08 on dist-test-slave-2sb1"
server_key: "5a4dcb4b0f42152cb6d47039ebae1d3d"
server_key_iv: "864c2f428e603b2dda65217e89d160af"
server_key_version: "encryptionkey@0"
I20260619 14:47:08.771776 8792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d0938919e2a248739b6179b3305cd623"
format_stamp: "Formatted at 2026-06-19 14:47:08 on dist-test-slave-2sb1"
server_key: "5a4dcb4b0f42152cb6d47039ebae1d3d"
server_key_iv: "864c2f428e603b2dda65217e89d160af"
server_key_version: "encryptionkey@0"
I20260619 14:47:08.776077 8792 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 14:47:08.779004 8807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:47:08.780902 8792 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:47:08.781080 8792 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/ts-1/wal
uuid: "d0938919e2a248739b6179b3305cd623"
format_stamp: "Formatted at 2026-06-19 14:47:08 on dist-test-slave-2sb1"
server_key: "5a4dcb4b0f42152cb6d47039ebae1d3d"
server_key_iv: "864c2f428e603b2dda65217e89d160af"
server_key_version: "encryptionkey@0"
I20260619 14:47:08.781225 8792 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:47:08.791684 8792 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:47:08.796144 8792 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:47:08.796378 8792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:47:08.797114 8792 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:47:08.798022 8792 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:47:08.798072 8792 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:47:08.798115 8792 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:47:08.798166 8792 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:47:08.807046 8792 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39085
I20260619 14:47:08.807314 8920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39085 every 8 connection(s)
I20260619 14:47:08.808118 8792 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/ts-1/data/info.pb
I20260619 14:47:08.813632 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 8792
I20260619 14:47:08.813799 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/ts-1/wal/instance
I20260619 14:47:08.814074 392 external_mini_cluster.cc:1428] Setting key 7067e16125683f069cfe5a13c1843717
Jun 19 14:47:08 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880428, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 14:47:08.826205 8635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:08.810252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37081 (local address 127.0.98.62:43861)
0619 14:47:08.810427 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:08.810432 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:08.811683 (+ 1251us) server_negotiation.cc:407] Connection header received
0619 14:47:08.813512 (+ 1829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:08.813516 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:08.813585 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:08.813685 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:08.815304 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:08.815975 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:08.818153 (+ 2178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:08.818408 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:08.818545 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:08.821907 (+ 3362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:47:08.821910 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:47:08.821912 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:47:08.821940 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:47:08.823721 (+ 1781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:08.824364 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:08.824367 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:08.824368 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:08.824452 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:08.824946 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:08.824950 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:08.824952 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:08.825151 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:47:08.825260 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:08.825898 (+ 638us) server_negotiation.cc:299] Negotiation successful
0619 14:47:08.826030 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 14:47:08.827240 8923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:08.810797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43861 (local address 127.0.98.2:37081)
0619 14:47:08.811494 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:08.811559 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:08.813197 (+ 1638us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:08.813929 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:08.813940 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:08.814489 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:47:08.815145 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:08.815163 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:08.816203 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:08.816209 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:08.817971 (+ 1762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:08.817984 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:08.818597 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:08.818602 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:08.818884 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:08.819662 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:47:08.819691 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:47:08.821677 (+ 1986us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:47:08.823882 (+ 2205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:47:08.823890 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:47:08.823910 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:47:08.824233 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:47:08.824553 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:47:08.824556 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:47:08.824558 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:47:08.824829 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:47:08.825273 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:47:08.825282 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:47:08.825567 (+ 285us) client_negotiation.cc:769] Sending connection context
0619 14:47:08.825834 (+ 267us) client_negotiation.cc:240] Negotiation successful
0619 14:47:08.826202 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":197,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260619 14:47:08.828805 8921 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43861
I20260619 14:47:08.829099 8921 heartbeater.cc:464] Registering TS with master...
I20260619 14:47:08.829804 8921 heartbeater.cc:511] Master 127.0.98.62:43861 requested a full tablet report, sending...
I20260619 14:47:08.831252 8595 ts_manager.cc:194] Registered new tserver with Master: d0938919e2a248739b6179b3305cd623 (127.0.98.2:39085)
I20260619 14:47:08.832110 8595 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:37081
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 14:47:08.850230 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43861
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:47:08.970461 8926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:47:08.970765 8926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:47:08.970813 8926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:47:08.975037 8926 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:47:08.975256 8926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:47:08.975330 8926 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 14:47:08.981834 8926 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43861
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8926
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:47:08.983023 8926 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:47:08.984036 8926 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:47:08.994881 8932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:47:08.995064 8926 server_base.cc:1034] running on GCE node
W20260619 14:47:08.994858 8934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:47:08.994923 8931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:47:08.995754 8926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:47:08.996605 8926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:47:08.997888 8926 hybrid_clock.cc:648] HybridClock initialized: now 1781880428997818 us; error 90 us; skew 500 ppm
Jun 19 14:47:08 dist-test-slave-2sb1 krb5kdc[8551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880428, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:47:09.001497 8926 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 14:47:09.002635 8926 webserver.cc:460] Webserver started at http://127.0.98.3:37367/ using document root <none> and password file <none>
I20260619 14:47:09.003248 8926 fs_manager.cc:359] Metadata directory not provided
I20260619 14:47:09.003346 8926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:47:09.003587 8926 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:47:09.005949 8926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/ts-2/data/instance:
uuid: "16e6be613a684c4690922fd5c6b4fc64"
format_stamp: "Formatted at 2026-06-19 14:47:08 on dist-test-slave-2sb1"
server_key: "00aaf451113b81ca68e648fc912a957b"
server_key_iv: "c4f7ff2dd3ec8fc27b694d037e3c6bae"
server_key_version: "encryptionkey@0"
I20260619 14:47:09.006537 8926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/ts-2/wal/instance:
uuid: "16e6be613a684c4690922fd5c6b4fc64"
format_stamp: "Formatted at 2026-06-19 14:47:08 on dist-test-slave-2sb1"
server_key: "00aaf451113b81ca68e648fc912a957b"
server_key_iv: "c4f7ff2dd3ec8fc27b694d037e3c6bae"
server_key_version: "encryptionkey@0"
I20260619 14:47:09.011173 8926 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260619 14:47:09.014184 8941 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:47:09.015694 8926 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 14:47:09.015875 8926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/ts-2/wal
uuid: "16e6be613a684c4690922fd5c6b4fc64"
format_stamp: "Formatted at 2026-06-19 14:47:08 on dist-test-slave-2sb1"
server_key: "00aaf451113b81ca68e648fc912a957b"
server_key_iv: "c4f7ff2dd3ec8fc27b694d037e3c6bae"
server_key_version: "encryptionkey@0"
I20260619 14:47:09.016022 8926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:47:09.034786 8926 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:47:09.038980 8926 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:47:09.039227 8926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:47:09.039882 8926 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:47:09.040828 8926 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:47:09.040897 8926 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:47:09.040966 8926 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:47:09.041014 8926 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:47:09.050932 8926 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43391
I20260619 14:47:09.051005 9054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43391 every 8 connection(s)
I20260619 14:47:09.052282 8926 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/ts-2/data/info.pb
I20260619 14:47:09.058368 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 8926
I20260619 14:47:09.058522 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/ts-2/wal/instance
I20260619 14:47:09.058825 392 external_mini_cluster.cc:1428] Setting key 2a80de7b3b11abe042cc62d6bb00bf51
Jun 19 14:47:09 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880428, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:47:09.069491 8635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:09.054676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59587 (local address 127.0.98.62:43861)
0619 14:47:09.054919 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:09.054925 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:09.056182 (+ 1257us) server_negotiation.cc:407] Connection header received
0619 14:47:09.058148 (+ 1966us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:09.058153 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:09.058216 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:09.058274 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:09.060238 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:09.060879 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:09.062353 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:09.062570 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:09.062678 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:09.065587 (+ 2909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:47:09.065591 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:47:09.065593 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:47:09.065620 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:47:09.067229 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:09.067822 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:09.067827 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:09.067828 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:09.067914 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:09.068299 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:09.068303 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:09.068305 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:09.068500 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:47:09.068608 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:09.069176 (+ 568us) server_negotiation.cc:299] Negotiation successful
0619 14:47:09.069312 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134}
I20260619 14:47:09.070331 9057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:09.055201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43861 (local address 127.0.98.3:59587)
0619 14:47:09.055975 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:09.056057 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:09.057850 (+ 1793us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:09.058770 (+ 920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:09.058784 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:09.059371 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:47:09.060089 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:09.060105 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:09.061042 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:09.061045 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:09.062210 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:09.062220 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:09.062745 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:09.062748 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:09.062900 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:09.063567 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:47:09.063585 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:47:09.065364 (+ 1779us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:47:09.067374 (+ 2010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:47:09.067381 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:47:09.067400 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:47:09.067703 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:47:09.068023 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:47:09.068026 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:47:09.068028 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:47:09.068193 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:47:09.068607 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:47:09.068614 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:47:09.068918 (+ 304us) client_negotiation.cc:769] Sending connection context
0619 14:47:09.069124 (+ 206us) client_negotiation.cc:240] Negotiation successful
0619 14:47:09.069469 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":131,"threads_started":1}
I20260619 14:47:09.071923 9055 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43861
I20260619 14:47:09.072332 9055 heartbeater.cc:464] Registering TS with master...
I20260619 14:47:09.073025 9055 heartbeater.cc:511] Master 127.0.98.62:43861 requested a full tablet report, sending...
I20260619 14:47:09.074077 8595 ts_manager.cc:194] Registered new tserver with Master: 16e6be613a684c4690922fd5c6b4fc64 (127.0.98.3:43391)
I20260619 14:47:09.074747 8595 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59587
I20260619 14:47:09.086472 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260619 14:47:09.180831 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 8658
I20260619 14:47:09.186728 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 8792
I20260619 14:47:09.192443 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 8926
I20260619 14:47:09.198383 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 8564
I20260619 14:47:09.204943 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43861
--webserver_interface=127.0.98.62
--webserver_port=45907
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43861
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:47:09.328773 9064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:47:09.329139 9064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:47:09.329226 9064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:47:09.333523 9064 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:47:09.333640 9064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:47:09.333678 9064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:47:09.333716 9064 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:47:09.333765 9064 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:47:09.341617 9064 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43861
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=45907
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9064
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:47:09.343063 9064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:47:09.344386 9064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:47:09.353842 9070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:47:09.354050 9069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:47:09.354298 9072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:47:09.354357 9064 server_base.cc:1034] running on GCE node
I20260619 14:47:09.355149 9064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:47:09.356544 9064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:47:09.357951 9064 hybrid_clock.cc:648] HybridClock initialized: now 1781880429357924 us; error 51 us; skew 500 ppm
Jun 19 14:47:09 dist-test-slave-2sb1 krb5kdc[8551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880429, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:47:09.361515 9064 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 14:47:09.362655 9064 webserver.cc:460] Webserver started at http://127.0.98.62:45907/ using document root <none> and password file <none>
I20260619 14:47:09.363270 9064 fs_manager.cc:359] Metadata directory not provided
I20260619 14:47:09.363389 9064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:47:09.367755 9064 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 14:47:09.370524 9079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:47:09.373596 9064 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 14:47:09.373862 9064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/master-0/wal
uuid: "457c5d49e6e9463199d5eb8401fae9db"
format_stamp: "Formatted at 2026-06-19 14:47:08 on dist-test-slave-2sb1"
server_key: "d87066ea5b37bc0006a0dd0cceb559a2"
server_key_iv: "2909e8c47769e2300dba18252402a44d"
server_key_version: "encryptionkey@0"
I20260619 14:47:09.374743 9064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:47:09.391337 9064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:47:09.395336 9064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:47:09.395582 9064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:47:09.403766 9064 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43861
I20260619 14:47:09.404047 9131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43861 every 8 connection(s)
I20260619 14:47:09.405056 9064 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:47:09.410187 9132 sys_catalog.cc:263] Verifying existing consensus state
I20260619 14:47:09.412117 9132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 457c5d49e6e9463199d5eb8401fae9db: Bootstrap starting.
I20260619 14:47:09.412823 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 9064
I20260619 14:47:09.413355 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:46849
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:43861
--webserver_port=41491
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:47:09.432163 9132 log.cc:824] T 00000000000000000000000000000000 P 457c5d49e6e9463199d5eb8401fae9db: Log is configured to *not* fsync() on all Append() calls
I20260619 14:47:09.439953 9132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 457c5d49e6e9463199d5eb8401fae9db: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 14:47:09.440400 9132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 457c5d49e6e9463199d5eb8401fae9db: Bootstrap complete.
I20260619 14:47:09.444916 9132 raft_consensus.cc:348] T 00000000000000000000000000000000 P 457c5d49e6e9463199d5eb8401fae9db [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "457c5d49e6e9463199d5eb8401fae9db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43861 } }
I20260619 14:47:09.445356 9132 raft_consensus.cc:729] T 00000000000000000000000000000000 P 457c5d49e6e9463199d5eb8401fae9db [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 457c5d49e6e9463199d5eb8401fae9db, State: Initialized, Role: FOLLOWER
I20260619 14:47:09.445865 9132 consensus_queue.cc:260] T 00000000000000000000000000000000 P 457c5d49e6e9463199d5eb8401fae9db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "457c5d49e6e9463199d5eb8401fae9db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43861 } }
I20260619 14:47:09.445978 9132 raft_consensus.cc:388] T 00000000000000000000000000000000 P 457c5d49e6e9463199d5eb8401fae9db [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:47:09.446066 9132 raft_consensus.cc:482] T 00000000000000000000000000000000 P 457c5d49e6e9463199d5eb8401fae9db [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:47:09.446167 9132 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 457c5d49e6e9463199d5eb8401fae9db [term 1 FOLLOWER]: Advancing to term 2
I20260619 14:47:09.448447 9132 raft_consensus.cc:504] T 00000000000000000000000000000000 P 457c5d49e6e9463199d5eb8401fae9db [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "457c5d49e6e9463199d5eb8401fae9db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43861 } }
I20260619 14:47:09.448907 9132 leader_election.cc:302] T 00000000000000000000000000000000 P 457c5d49e6e9463199d5eb8401fae9db [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 457c5d49e6e9463199d5eb8401fae9db; no voters:
I20260619 14:47:09.449293 9132 leader_election.cc:288] T 00000000000000000000000000000000 P 457c5d49e6e9463199d5eb8401fae9db [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 14:47:09.449393 9136 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 457c5d49e6e9463199d5eb8401fae9db [term 2 FOLLOWER]: Leader election won for term 2
I20260619 14:47:09.449591 9136 raft_consensus.cc:686] T 00000000000000000000000000000000 P 457c5d49e6e9463199d5eb8401fae9db [term 2 LEADER]: Becoming Leader. State: Replica: 457c5d49e6e9463199d5eb8401fae9db, State: Running, Role: LEADER
I20260619 14:47:09.449903 9136 consensus_queue.cc:237] T 00000000000000000000000000000000 P 457c5d49e6e9463199d5eb8401fae9db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "457c5d49e6e9463199d5eb8401fae9db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43861 } }
I20260619 14:47:09.450760 9132 sys_catalog.cc:564] T 00000000000000000000000000000000 P 457c5d49e6e9463199d5eb8401fae9db [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:47:09.451520 9137 sys_catalog.cc:455] T 00000000000000000000000000000000 P 457c5d49e6e9463199d5eb8401fae9db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "457c5d49e6e9463199d5eb8401fae9db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "457c5d49e6e9463199d5eb8401fae9db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43861 } } }
I20260619 14:47:09.451665 9137 sys_catalog.cc:458] T 00000000000000000000000000000000 P 457c5d49e6e9463199d5eb8401fae9db [sys.catalog]: This master's current role is: LEADER
I20260619 14:47:09.452009 9138 sys_catalog.cc:455] T 00000000000000000000000000000000 P 457c5d49e6e9463199d5eb8401fae9db [sys.catalog]: SysCatalogTable state changed. Reason: New leader 457c5d49e6e9463199d5eb8401fae9db. Latest consensus state: current_term: 2 leader_uuid: "457c5d49e6e9463199d5eb8401fae9db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "457c5d49e6e9463199d5eb8401fae9db" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43861 } } }
I20260619 14:47:09.452212 9138 sys_catalog.cc:458] T 00000000000000000000000000000000 P 457c5d49e6e9463199d5eb8401fae9db [sys.catalog]: This master's current role is: LEADER
I20260619 14:47:09.452497 9140 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:47:09.456435 9140 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:47:09.457242 9140 catalog_manager.cc:1259] Loaded cluster ID: e58eca3524b9443e91ba5804a523970a
I20260619 14:47:09.457981 9140 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:47:09.460745 9140 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:47:09.461517 9140 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 457c5d49e6e9463199d5eb8401fae9db: Loaded TSK: 0
I20260619 14:47:09.462792 9140 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 14:47:09.543680 9134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:47:09.544014 9134 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:47:09.544096 9134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:47:09.548429 9134 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:47:09.548549 9134 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:47:09.548699 9134 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 14:47:09.554927 9134 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:46849
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=41491
--tserver_master_addrs=127.0.98.62:43861
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9134
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:47:09.556010 9134 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:47:09.557308 9134 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:47:09.567807 9158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:47:09.567636 9159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:47:09.567641 9161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:47:09.568230 9134 server_base.cc:1034] running on GCE node
I20260619 14:47:09.568912 9134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:47:09.569782 9134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:47:09.570984 9134 hybrid_clock.cc:648] HybridClock initialized: now 1781880429570944 us; error 54 us; skew 500 ppm
Jun 19 14:47:09 dist-test-slave-2sb1 krb5kdc[8551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880429, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:47:09.574692 9134 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260619 14:47:09.576004 9134 webserver.cc:460] Webserver started at http://127.0.98.1:41491/ using document root <none> and password file <none>
I20260619 14:47:09.576828 9134 fs_manager.cc:359] Metadata directory not provided
I20260619 14:47:09.576948 9134 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:47:09.583007 9134 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 14:47:09.586246 9168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:47:09.588061 9134 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260619 14:47:09.588249 9134 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/ts-0/wal
uuid: "e7c951ef1c0b4804a75f903a9d22a229"
format_stamp: "Formatted at 2026-06-19 14:47:08 on dist-test-slave-2sb1"
server_key: "4b7cd6af151488754bb5614d0a01f96a"
server_key_iv: "18774166599f14831c17fc316d6b0b09"
server_key_version: "encryptionkey@0"
I20260619 14:47:09.589148 9134 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:47:09.607436 9134 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:47:09.611603 9134 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:47:09.611845 9134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:47:09.612587 9134 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:47:09.613900 9134 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:47:09.613971 9134 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:47:09.614028 9134 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:47:09.614054 9134 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:47:09.625640 9134 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46849
I20260619 14:47:09.625816 9281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46849 every 8 connection(s)
I20260619 14:47:09.627009 9134 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/ts-0/data/info.pb
I20260619 14:47:09.632103 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 9134
I20260619 14:47:09.632602 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:39085
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:43861
--webserver_port=40701
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 14:47:09 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:47:09 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:47:09.644994 9285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:09.629903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43861 (local address 127.0.98.1:56043)
0619 14:47:09.630417 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:09.630462 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:09.631820 (+ 1358us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:09.633446 (+ 1626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:09.633458 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:09.634017 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:47:09.634630 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:09.634657 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:09.636474 (+ 1817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:09.636481 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:09.638380 (+ 1899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:09.638401 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:09.639520 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:09.639525 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:09.639814 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:09.640566 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:47:09.640596 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:47:09.643770 (+ 3174us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43861: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":164,"threads_started":1}
W20260619 14:47:09.644984 9284 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:47:09.629698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56043 (local address 127.0.98.62:43861)
0619 14:47:09.630190 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:09.630214 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:09.630591 (+ 377us) server_negotiation.cc:407] Connection header received
0619 14:47:09.632365 (+ 1774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:09.632376 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:09.632961 (+ 585us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:09.633262 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:09.634880 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:09.636275 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:09.638729 (+ 2454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:09.639300 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:09.639451 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:09.644034 (+ 4583us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:56043: BlockingRecv error: recv got EOF from 127.0.98.1:56043 (error 108)
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":142,"threads_started":1}
W20260619 14:47:09.645599 9282 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43861 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:43861: Client connection negotiation failed: client connection to 127.0.98.62:43861: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Jun 19 14:47:09 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:47:09 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:47:09.652981 9285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:09.646150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43861 (local address 127.0.98.1:58493)
0619 14:47:09.646344 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:09.646367 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:09.646451 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:09.646773 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:09.646775 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:09.646927 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:47:09.647038 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:09.647043 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:09.648107 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:09.648110 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:09.648925 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:09.648937 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:09.649532 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:09.649535 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:09.649614 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:09.650343 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:47:09.650355 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:47:09.652836 (+ 2481us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43861: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":96}
W20260619 14:47:09.653559 9284 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:47:09.646283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58493 (local address 127.0.98.62:43861)
0619 14:47:09.646421 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:09.646426 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:09.646443 (+ 17us) server_negotiation.cc:407] Connection header received
0619 14:47:09.646531 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:09.646534 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:09.646611 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:09.646700 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:09.647146 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:09.647967 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:09.649043 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:09.649405 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:09.649511 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:09.653092 (+ 3581us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:58493: BlockingRecv error: recv got EOF from 127.0.98.1:58493 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 19 14:47:09 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:47:09 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:47:09.659963 9285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:09.653637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43861 (local address 127.0.98.1:38591)
0619 14:47:09.653801 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:09.653815 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:09.653890 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:09.654148 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:09.654151 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:09.654328 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:47:09.654445 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:09.654451 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:09.655453 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:09.655456 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:09.656340 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:09.656351 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:09.657009 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:09.657013 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:09.657105 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:09.657635 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:47:09.657650 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:47:09.659818 (+ 2168us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43861: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":97}
W20260619 14:47:09.660418 9284 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:47:09.653745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38591 (local address 127.0.98.62:43861)
0619 14:47:09.653874 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:09.653879 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:09.653895 (+ 16us) server_negotiation.cc:407] Connection header received
0619 14:47:09.653996 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:09.653998 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:09.654041 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:09.654108 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:09.654577 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:09.655328 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:09.656480 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:09.656877 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:09.656988 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:09.659991 (+ 3003us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:38591: BlockingRecv error: recv got EOF from 127.0.98.1:38591 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260619 14:47:09.660422 9282 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 14:47:09.759274 9286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:47:09.759634 9286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:47:09.759712 9286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:47:09.764887 9286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:47:09.764991 9286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:47:09.765089 9286 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 14:47:09.770838 9286 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:39085
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=40701
--tserver_master_addrs=127.0.98.62:43861
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9286
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:47:09.771950 9286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:47:09.773061 9286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:47:09.783491 9291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:47:09.783514 9294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:47:09.783514 9292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:47:09.784011 9286 server_base.cc:1034] running on GCE node
I20260619 14:47:09.784480 9286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:47:09.785389 9286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:47:09.786613 9286 hybrid_clock.cc:648] HybridClock initialized: now 1781880429786562 us; error 61 us; skew 500 ppm
Jun 19 14:47:09 dist-test-slave-2sb1 krb5kdc[8551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880429, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:47:09.790253 9286 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260619 14:47:09.791460 9286 webserver.cc:460] Webserver started at http://127.0.98.2:40701/ using document root <none> and password file <none>
I20260619 14:47:09.792075 9286 fs_manager.cc:359] Metadata directory not provided
I20260619 14:47:09.792194 9286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:47:09.797041 9286 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:47:09.799746 9301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:47:09.801468 9286 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:47:09.801723 9286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/ts-1/wal
uuid: "d0938919e2a248739b6179b3305cd623"
format_stamp: "Formatted at 2026-06-19 14:47:08 on dist-test-slave-2sb1"
server_key: "5a4dcb4b0f42152cb6d47039ebae1d3d"
server_key_iv: "864c2f428e603b2dda65217e89d160af"
server_key_version: "encryptionkey@0"
I20260619 14:47:09.802419 9286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:47:09.815289 9286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:47:09.819074 9286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:47:09.819406 9286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:47:09.820026 9286 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:47:09.821029 9286 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:47:09.821215 9286 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:47:09.821307 9286 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:47:09.821367 9286 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:47:09.831398 9286 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39085
I20260619 14:47:09.831416 9414 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39085 every 8 connection(s)
I20260619 14:47:09.832477 9286 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/ts-1/data/info.pb
I20260619 14:47:09.841444 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 9286
I20260619 14:47:09.842008 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:43391
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:43861
--webserver_port=37367
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 14:47:09 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:47:09 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 14:47:09.845570 9284 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:47:09.834717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55125 (local address 127.0.98.62:43861)
0619 14:47:09.834902 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:09.834907 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:09.835923 (+ 1016us) server_negotiation.cc:407] Connection header received
0619 14:47:09.837397 (+ 1474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:09.837400 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:09.837456 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:09.837535 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:09.838874 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:09.839454 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:09.840774 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:09.840985 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:09.841069 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:09.845373 (+ 4304us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:55125: BlockingRecv error: recv got EOF from 127.0.98.2:55125 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
I20260619 14:47:09.846170 9417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:09.835223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43861 (local address 127.0.98.2:55125)
0619 14:47:09.835746 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:09.835791 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:09.837152 (+ 1361us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:09.837828 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:09.837836 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:09.838222 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:47:09.838723 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:09.838737 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:09.839587 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:09.839591 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:09.840594 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:09.840604 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:09.841146 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:09.841149 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:09.841314 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:09.842115 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:47:09.842165 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:47:09.845164 (+ 2999us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43861: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":125,"threads_started":1}
W20260619 14:47:09.846843 9415 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43861 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:43861: Client connection negotiation failed: client connection to 127.0.98.62:43861: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Jun 19 14:47:09 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:47:09 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:47:09.854064 9417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:09.847488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43861 (local address 127.0.98.2:51823)
0619 14:47:09.847614 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:09.847631 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:09.847752 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:09.848041 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:09.848043 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:09.848203 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:47:09.848303 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:09.848308 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:09.849511 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:09.849514 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:09.850180 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:09.850189 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:09.850677 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:09.850680 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:09.850772 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:09.851412 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:47:09.851435 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:47:09.853859 (+ 2424us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43861: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":51}
W20260619 14:47:09.854257 9284 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:47:09.847592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51823 (local address 127.0.98.62:43861)
0619 14:47:09.847752 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:09.847759 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:09.847780 (+ 21us) server_negotiation.cc:407] Connection header received
0619 14:47:09.847835 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:09.847840 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:09.847900 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:09.848007 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:09.848420 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:09.849388 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:09.850334 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:09.850550 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:09.850668 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:09.854020 (+ 3352us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:51823: BlockingRecv error: recv got EOF from 127.0.98.2:51823 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 19 14:47:09 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:47:09 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:47:09.860332 9417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:09.854939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43861 (local address 127.0.98.2:38661)
0619 14:47:09.855066 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:09.855081 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:09.855169 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:09.855423 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:09.855426 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:09.855620 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:47:09.855719 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:09.855724 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:09.856593 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:09.856599 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:09.857227 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:09.857235 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:09.857743 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:09.857747 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:09.857828 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:09.858355 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:47:09.858370 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:47:09.860173 (+ 1803us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43861: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":54}
W20260619 14:47:09.860515 9284 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:47:09.855004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38661 (local address 127.0.98.62:43861)
0619 14:47:09.855163 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:09.855168 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:09.855183 (+ 15us) server_negotiation.cc:407] Connection header received
0619 14:47:09.855271 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:09.855274 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:09.855322 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:09.855407 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:09.855837 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:09.856457 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:09.857382 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:09.857623 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:09.857739 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:09.860293 (+ 2554us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:38661: BlockingRecv error: recv got EOF from 127.0.98.2:38661 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
W20260619 14:47:09.860745 9415 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 14:47:09.962108 9418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:47:09.962417 9418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:47:09.962466 9418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:47:09.967163 9418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:47:09.967250 9418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:47:09.967319 9418 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 14:47:09.973006 9418 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:43391
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=37367
--tserver_master_addrs=127.0.98.62:43861
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9418
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:47:09.974124 9418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:47:09.975160 9418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:47:09.986547 9424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:47:09.986554 9426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:47:09.986565 9423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:47:09.986954 9418 server_base.cc:1034] running on GCE node
I20260619 14:47:09.987488 9418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:47:09.988206 9418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:47:09.989427 9418 hybrid_clock.cc:648] HybridClock initialized: now 1781880429989412 us; error 31 us; skew 500 ppm
Jun 19 14:47:09 dist-test-slave-2sb1 krb5kdc[8551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880429, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:47:09.993520 9418 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260619 14:47:09.994539 9418 webserver.cc:460] Webserver started at http://127.0.98.3:37367/ using document root <none> and password file <none>
I20260619 14:47:09.995247 9418 fs_manager.cc:359] Metadata directory not provided
I20260619 14:47:09.995365 9418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:47:10.000942 9418 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 14:47:10.003635 9433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:47:10.005419 9418 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:47:10.005550 9418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/ts-2/wal
uuid: "16e6be613a684c4690922fd5c6b4fc64"
format_stamp: "Formatted at 2026-06-19 14:47:08 on dist-test-slave-2sb1"
server_key: "00aaf451113b81ca68e648fc912a957b"
server_key_iv: "c4f7ff2dd3ec8fc27b694d037e3c6bae"
server_key_version: "encryptionkey@0"
I20260619 14:47:10.006346 9418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:47:10.024595 9418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:47:10.028388 9418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:47:10.028684 9418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:47:10.029333 9418 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:47:10.030210 9418 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:47:10.030292 9418 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:47:10.030359 9418 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:47:10.030412 9418 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:47:10.041306 9418 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43391
I20260619 14:47:10.041322 9546 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43391 every 8 connection(s)
I20260619 14:47:10.042414 9418 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781880383132139-392-0/minicluster-data/ts-2/data/info.pb
I20260619 14:47:10.050490 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 9418
Jun 19 14:47:10 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:47:10 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260619 14:47:10.057569 9284 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:47:10.044992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50321 (local address 127.0.98.62:43861)
0619 14:47:10.045163 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:10.045167 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:10.046058 (+ 891us) server_negotiation.cc:407] Connection header received
0619 14:47:10.047554 (+ 1496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:10.047558 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:10.047617 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:10.047896 (+ 279us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:10.049750 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:10.050323 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:10.051921 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:10.052254 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:10.052333 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:10.057417 (+ 5084us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:50321: BlockingRecv error: recv got EOF from 127.0.98.3:50321 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 14:47:10.058077 9549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:10.045378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43861 (local address 127.0.98.3:50321)
0619 14:47:10.045885 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:10.045930 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:10.047306 (+ 1376us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:10.048086 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:10.048094 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:10.048750 (+ 656us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:47:10.049419 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:10.049449 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:10.050481 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:10.050489 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:10.051745 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:10.051757 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:10.052419 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:10.052424 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:10.052847 (+ 423us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:10.053620 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:47:10.053638 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:47:10.057145 (+ 3507us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43861: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":139,"threads_started":1}
W20260619 14:47:10.058660 9547 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:43861 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:43861: Client connection negotiation failed: client connection to 127.0.98.62:43861: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
I20260619 14:47:10.062764 9551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:10.051661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57876 (local address 127.0.98.62:43861)
0619 14:47:10.051901 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:10.051906 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:10.051919 (+ 13us) server_negotiation.cc:407] Connection header received
0619 14:47:10.051948 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:10.051951 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:10.051994 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:10.052088 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:10.052999 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:10.053795 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:10.054924 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:10.055246 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:10.055378 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:10.057040 (+ 1662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:47:10.057046 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:47:10.057064 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:47:10.057094 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:47:10.059869 (+ 2775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:10.060778 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:10.060784 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:10.060790 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:10.060915 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:10.061330 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:10.061333 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:10.061334 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:10.061689 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:47:10.061936 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:10.062267 (+ 331us) server_negotiation.cc:299] Negotiation successful
0619 14:47:10.062550 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":82,"threads_started":1}
Jun 19 14:47:10 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:47:10 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:47:10.065138 9549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:10.059230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43861 (local address 127.0.98.3:35603)
0619 14:47:10.059414 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:10.059429 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:10.059522 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:10.059885 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:10.059888 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:10.060066 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:47:10.060163 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:10.060168 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:10.060955 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:10.060960 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:10.061609 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:10.061620 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:10.062076 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:10.062080 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:10.062173 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:10.062920 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:47:10.062940 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:47:10.065009 (+ 2069us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43861: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118}
W20260619 14:47:10.065429 9284 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:47:10.059314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35603 (local address 127.0.98.62:43861)
0619 14:47:10.059429 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:10.059432 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:10.059486 (+ 54us) server_negotiation.cc:407] Connection header received
0619 14:47:10.059626 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:10.059628 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:10.059666 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:10.059840 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:10.060287 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:10.060851 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:10.061751 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:10.061962 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:10.062073 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:10.065305 (+ 3232us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:35603: BlockingRecv error: recv got EOF from 127.0.98.3:35603 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Jun 19 14:47:10 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:47:10 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:47:10.071960 9549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:10.065875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43861 (local address 127.0.98.3:52579)
0619 14:47:10.066124 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:10.066137 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:10.066223 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:10.066655 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:10.066658 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:10.066832 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:47:10.066913 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:10.066918 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:10.067697 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:10.067702 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:10.068301 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:10.068311 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:10.068973 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:10.068977 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:10.069060 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:10.069595 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:47:10.069609 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:47:10.071852 (+ 2243us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43861: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":55}
W20260619 14:47:10.072232 9284 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:47:10.065924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52579 (local address 127.0.98.62:43861)
0619 14:47:10.066061 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:10.066065 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:10.066420 (+ 355us) server_negotiation.cc:407] Connection header received
0619 14:47:10.066471 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:10.066474 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:10.066524 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:10.066644 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:10.067030 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:10.067574 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:10.068428 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:10.068853 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:10.068970 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:10.072053 (+ 3083us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:52579: BlockingRecv error: recv got EOF from 127.0.98.3:52579 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260619 14:47:10.072501 9547 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 19 14:47:10 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:47:10 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:47:10.672963 9552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:10.661078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43861 (local address 127.0.98.1:59073)
0619 14:47:10.661446 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:10.661460 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:10.661580 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:10.662052 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:10.662054 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:10.662250 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:47:10.662372 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:10.662378 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:10.663324 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:10.663327 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:10.663961 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:10.663970 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:10.664565 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:10.664568 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:10.664814 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:10.670606 (+ 5792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:47:10.670620 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:47:10.672826 (+ 2206us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43861: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":84,"threads_started":1}
W20260619 14:47:10.673200 9553 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:47:10.661224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59073 (local address 127.0.98.62:43861)
0619 14:47:10.661445 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:10.661448 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:10.661579 (+ 131us) server_negotiation.cc:407] Connection header received
0619 14:47:10.661846 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:10.661848 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:10.661898 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:10.661987 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:10.662513 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:10.663165 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:10.664109 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:10.664442 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:10.664537 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:10.673027 (+ 8490us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:59073: BlockingRecv error: recv got EOF from 127.0.98.1:59073 (error 108)
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":58,"threads_started":1}
Jun 19 14:47:10 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:47:10 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:47:10.868386 9554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:10.861623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43861 (local address 127.0.98.2:50119)
0619 14:47:10.861997 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:10.862012 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:10.862130 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:10.862387 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:10.862389 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:10.862580 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:47:10.862789 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:10.862797 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:10.863645 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:10.863648 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:10.864342 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:10.864352 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:10.864972 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:10.864975 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:10.865063 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:10.865692 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:47:10.865781 (+ 89us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:47:10.868116 (+ 2335us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43861: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":98,"threads_started":1}
W20260619 14:47:10.868383 9553 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:47:10.861728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50119 (local address 127.0.98.62:43861)
0619 14:47:10.861961 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:10.861970 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:10.862102 (+ 132us) server_negotiation.cc:407] Connection header received
0619 14:47:10.862205 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:10.862207 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:10.862259 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:10.862348 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:10.862976 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:10.863521 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:10.864483 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:10.864754 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:10.864930 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:10.868271 (+ 3341us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:50119: BlockingRecv error: recv got EOF from 127.0.98.2:50119 (error 108)
Metrics: {"server-negotiator.queue_time_us":132}
Jun 19 14:47:11 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:47:11 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:47:11.080039 9555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:11.073502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43861 (local address 127.0.98.3:48541)
0619 14:47:11.073758 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:11.073771 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:11.073874 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:11.074136 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:11.074139 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:11.074420 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:47:11.074542 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:11.074548 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:11.075360 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:11.075367 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:11.075935 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:11.075944 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:11.076426 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:11.076429 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:11.076556 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:11.077168 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:47:11.077190 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:47:11.079875 (+ 2685us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43861: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":80,"threads_started":1}
W20260619 14:47:11.080171 9553 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:47:11.073549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48541 (local address 127.0.98.62:43861)
0619 14:47:11.073676 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:11.073681 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:11.073830 (+ 149us) server_negotiation.cc:407] Connection header received
0619 14:47:11.073948 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:11.073951 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:11.074009 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:11.074097 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:11.074664 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:11.075199 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:11.076062 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:11.076305 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:11.076420 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:11.080036 (+ 3616us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:48541: BlockingRecv error: recv got EOF from 127.0.98.3:48541 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Jun 19 14:47:11 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:47:11 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:47:11.680496 9556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:11.674006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43861 (local address 127.0.98.1:35227)
0619 14:47:11.674296 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:11.674311 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:11.674403 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:11.674825 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:11.674828 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:11.675027 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:47:11.675126 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:11.675132 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:11.675944 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:11.675947 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:11.676627 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:11.676636 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:11.677149 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:11.677152 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:11.677218 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:11.677923 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:47:11.677945 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:47:11.680358 (+ 2413us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43861: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":85,"threads_started":1}
W20260619 14:47:11.680827 9557 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:47:11.674113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35227 (local address 127.0.98.62:43861)
0619 14:47:11.674436 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:11.674439 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:11.674453 (+ 14us) server_negotiation.cc:407] Connection header received
0619 14:47:11.674492 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:11.674494 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:11.674535 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:11.674613 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:11.675310 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:11.675836 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:11.676800 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:11.677022 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:11.677105 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:11.680520 (+ 3415us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:35227: BlockingRecv error: recv got EOF from 127.0.98.1:35227 (error 108)
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":174,"threads_started":1}
Jun 19 14:47:11 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:47:11 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:47:11.876241 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:11.869472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43861 (local address 127.0.98.2:38179)
0619 14:47:11.870008 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:11.870030 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:11.870182 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:11.870488 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:11.870491 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:11.870712 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:47:11.870805 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:11.870811 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:11.871667 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:11.871671 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:11.872290 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:11.872299 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:11.872855 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:11.872859 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:11.872947 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:11.873572 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:47:11.873585 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:47:11.876021 (+ 2436us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43861: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":92,"threads_started":1}
W20260619 14:47:11.876252 9557 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:47:11.869631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38179 (local address 127.0.98.62:43861)
0619 14:47:11.869970 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:11.869975 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:11.870121 (+ 146us) server_negotiation.cc:407] Connection header received
0619 14:47:11.870310 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:11.870314 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:11.870370 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:11.870467 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:11.870938 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:11.871529 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:11.872421 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:11.872720 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:11.872847 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:11.876152 (+ 3305us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:38179: BlockingRecv error: recv got EOF from 127.0.98.2:38179 (error 108)
Metrics: {"server-negotiator.queue_time_us":238}
Jun 19 14:47:12 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:47:12 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:47:12.088230 9559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:12.081153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43861 (local address 127.0.98.3:33727)
0619 14:47:12.081547 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:12.081562 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:12.081676 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:12.082029 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:12.082032 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:12.082213 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:47:12.082337 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:12.082343 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:12.083395 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:12.083398 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:12.083968 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:12.083977 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:12.084693 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:12.084697 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:12.084791 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:12.085506 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:47:12.085518 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:47:12.088076 (+ 2558us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43861: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":129,"threads_started":1}
W20260619 14:47:12.088491 9557 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:47:12.081430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33727 (local address 127.0.98.62:43861)
0619 14:47:12.081612 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:12.081765 (+ 153us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:12.081785 (+ 20us) server_negotiation.cc:407] Connection header received
0619 14:47:12.081830 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:12.081832 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:12.081880 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:12.081963 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:12.082485 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:12.083170 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:12.084213 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:12.084489 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:12.084604 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:12.088256 (+ 3652us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:33727: BlockingRecv error: recv got EOF from 127.0.98.3:33727 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Jun 19 14:47:12 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:47:12 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:47:12.688215 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:12.681416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43861 (local address 127.0.98.1:59263)
0619 14:47:12.681710 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:12.681734 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:12.681833 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:12.682145 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:12.682147 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:12.682347 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:47:12.682476 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:12.682484 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:12.683397 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:12.683400 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:12.684146 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:12.684160 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:12.684842 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:12.684846 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:12.684919 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:12.685523 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:47:12.685536 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:47:12.688000 (+ 2464us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43861: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":119,"threads_started":1}
W20260619 14:47:12.688279 9561 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:47:12.681559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59263 (local address 127.0.98.62:43861)
0619 14:47:12.681805 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:12.681809 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:12.681825 (+ 16us) server_negotiation.cc:407] Connection header received
0619 14:47:12.681907 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:12.681909 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:12.681962 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:12.682062 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:12.682628 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:12.683245 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:12.684305 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:12.684542 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:12.684639 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:12.688146 (+ 3507us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:59263: BlockingRecv error: recv got EOF from 127.0.98.1:59263 (error 108)
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":109,"threads_started":1}
Jun 19 14:47:12 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:47:12 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:47:12.884106 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:12.877241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43861 (local address 127.0.98.2:38191)
0619 14:47:12.877612 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:12.877628 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:12.877853 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:12.878208 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:12.878211 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:12.878428 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:47:12.878517 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:12.878523 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:12.879779 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:12.879782 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:12.880442 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:12.880452 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:12.881034 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:12.881038 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:12.881119 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:12.881802 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:47:12.881815 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:47:12.883911 (+ 2096us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43861: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":94,"threads_started":1}
W20260619 14:47:12.884177 9561 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:47:12.877409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38191 (local address 127.0.98.62:43861)
0619 14:47:12.877599 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:12.877604 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:12.877762 (+ 158us) server_negotiation.cc:407] Connection header received
0619 14:47:12.878018 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:12.878024 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:12.878081 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:12.878183 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:12.878689 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:12.879322 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:12.880572 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:12.880900 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:12.881031 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:12.884065 (+ 3034us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:38191: BlockingRecv error: recv got EOF from 127.0.98.2:38191 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Jun 19 14:47:13 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:47:13 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:47:13.097056 9563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:13.089536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43861 (local address 127.0.98.3:56589)
0619 14:47:13.090017 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:13.090036 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:13.090212 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:13.090530 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:13.090533 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:13.090898 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:47:13.091002 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:13.091009 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:13.091937 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:13.091942 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:13.092563 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:13.092572 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:13.093193 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:13.093197 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:13.093293 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:13.093921 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:47:13.093934 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:47:13.096900 (+ 2966us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43861: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":133,"threads_started":1}
W20260619 14:47:13.097182 9561 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:47:13.089723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56589 (local address 127.0.98.62:43861)
0619 14:47:13.090017 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:13.090024 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:13.090141 (+ 117us) server_negotiation.cc:407] Connection header received
0619 14:47:13.090337 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:13.090341 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:13.090402 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:13.090506 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:13.091242 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:13.091808 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:13.092795 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:13.093045 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:13.093184 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:13.097039 (+ 3855us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:56589: BlockingRecv error: recv got EOF from 127.0.98.3:56589 (error 108)
Metrics: {"server-negotiator.queue_time_us":192}
Jun 19 14:47:13 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:47:13 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:47:13.696509 9564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:13.689389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43861 (local address 127.0.98.1:39847)
0619 14:47:13.689872 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:13.689887 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:13.690012 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:13.690430 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:13.690433 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:13.690615 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:47:13.690712 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:13.690718 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:13.691698 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:13.691701 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:13.692383 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:13.692393 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:13.693094 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:13.693097 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:13.693250 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:13.694039 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:47:13.694051 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:47:13.696294 (+ 2243us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43861: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":97,"threads_started":1}
W20260619 14:47:13.696703 9565 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:47:13.689591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39847 (local address 127.0.98.62:43861)
0619 14:47:13.689886 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:13.689889 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:13.689952 (+ 63us) server_negotiation.cc:407] Connection header received
0619 14:47:13.690118 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:13.690121 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:13.690166 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:13.690281 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:13.690937 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:13.691544 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:13.692522 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:13.692945 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:13.693045 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:13.696520 (+ 3475us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:39847: BlockingRecv error: recv got EOF from 127.0.98.1:39847 (error 108)
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":68,"threads_started":1}
Jun 19 14:47:13 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:47:13 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:47:13.892349 9566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:13.885407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43861 (local address 127.0.98.2:59837)
0619 14:47:13.885795 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:13.885813 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:13.885943 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:13.886310 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:13.886313 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:13.886556 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:47:13.886916 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:13.886930 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:13.887803 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:13.887807 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:13.888549 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:13.888559 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:13.889245 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:13.889250 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:13.889333 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:13.889920 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:47:13.889932 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:47:13.892198 (+ 2266us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43861: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":98,"threads_started":1}
W20260619 14:47:13.892761 9565 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:47:13.885462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59837 (local address 127.0.98.62:43861)
0619 14:47:13.885632 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:13.885637 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:13.886000 (+ 363us) server_negotiation.cc:407] Connection header received
0619 14:47:13.886056 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:13.886060 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:13.886119 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:13.886229 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:13.887077 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:13.887648 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:13.888873 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:13.889104 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:13.889188 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:13.892607 (+ 3419us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:59837: BlockingRecv error: recv got EOF from 127.0.98.2:59837 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Jun 19 14:47:14 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:47:14 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:47:14.105698 9567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:14.098428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43861 (local address 127.0.98.3:51315)
0619 14:47:14.098842 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:14.098856 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:14.098974 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:14.099426 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:14.099428 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:14.099605 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:47:14.099718 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:14.099723 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:14.100973 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:14.100978 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:14.101804 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:14.101818 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:14.102561 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:14.102566 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:14.102646 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:14.103310 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:47:14.103323 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:47:14.105505 (+ 2182us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43861: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":112,"threads_started":1}
W20260619 14:47:14.105959 9565 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:47:14.098535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51315 (local address 127.0.98.62:43861)
0619 14:47:14.098826 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:14.098831 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:14.098938 (+ 107us) server_negotiation.cc:407] Connection header received
0619 14:47:14.099146 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:14.099153 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:14.099232 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:14.099361 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:14.100019 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:14.100635 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:14.102018 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:14.102403 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:14.102518 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:14.105820 (+ 3302us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:51315: BlockingRecv error: recv got EOF from 127.0.98.3:51315 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Jun 19 14:47:14 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:47:14 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:47:14.704900 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:14.697588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43861 (local address 127.0.98.1:34165)
0619 14:47:14.697965 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:14.697981 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:14.698071 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:14.698446 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:14.698449 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:14.698658 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:47:14.698753 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:14.698759 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:14.699573 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:14.699577 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:14.700417 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:14.700430 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:14.701138 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:14.701141 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:14.701208 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:14.701869 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:47:14.701881 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:47:14.704505 (+ 2624us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43861: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":79,"threads_started":1}
W20260619 14:47:14.705250 9569 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:47:14.697738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34165 (local address 127.0.98.62:43861)
0619 14:47:14.698024 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:14.698028 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:14.698044 (+ 16us) server_negotiation.cc:407] Connection header received
0619 14:47:14.698212 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:14.698214 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:14.698268 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:14.698354 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:14.698893 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:14.699450 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:14.700578 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:14.701014 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:14.701109 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:14.704897 (+ 3788us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:34165: BlockingRecv error: recv got EOF from 127.0.98.1:34165 (error 108)
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":81,"threads_started":1}
Jun 19 14:47:14 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:47:14 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:47:14.905634 9570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:14.893529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43861 (local address 127.0.98.2:59519)
0619 14:47:14.893841 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:14.893862 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:14.893958 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:14.894254 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:14.894256 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:14.894436 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:47:14.894558 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:14.894619 (+ 61us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:14.895549 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:14.895552 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:14.896327 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:14.896337 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:14.896989 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:14.896993 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:14.897058 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:14.903197 (+ 6139us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:47:14.903211 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:47:14.905505 (+ 2294us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43861: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":92,"threads_started":1}
W20260619 14:47:14.905761 9569 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:47:14.893777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59519 (local address 127.0.98.62:43861)
0619 14:47:14.893918 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:14.893922 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:14.893936 (+ 14us) server_negotiation.cc:407] Connection header received
0619 14:47:14.894043 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:14.894046 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:14.894091 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:14.894164 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:14.894744 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:14.895423 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:14.896439 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:14.896714 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:14.896896 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:14.905633 (+ 8737us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:59519: BlockingRecv error: recv got EOF from 127.0.98.2:59519 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 19 14:47:15 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:47:15 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:47:15.113045 9571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:15.106694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43861 (local address 127.0.98.3:43543)
0619 14:47:15.106960 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:15.106975 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:15.107095 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:15.107420 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:15.107423 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:15.107605 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:47:15.107702 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:15.107708 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:15.108477 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:15.108481 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:15.109291 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:15.109302 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:15.109835 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:15.109839 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:15.109904 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:15.110699 (+ 795us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:47:15.110716 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:47:15.112918 (+ 2202us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43861: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":112,"threads_started":1}
W20260619 14:47:15.113289 9569 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:47:15.106771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43543 (local address 127.0.98.62:43861)
0619 14:47:15.106990 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:15.106995 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:15.107096 (+ 101us) server_negotiation.cc:407] Connection header received
0619 14:47:15.107189 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:15.107192 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:15.107240 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:15.107349 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:15.107813 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:15.108360 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:15.109440 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:15.109684 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:15.109798 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:15.113143 (+ 3345us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:43543: BlockingRecv error: recv got EOF from 127.0.98.3:43543 (error 108)
Metrics: {"server-negotiator.queue_time_us":140}
Jun 19 14:47:15 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:47:15 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:47:15.721920 9572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:15.706103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43861 (local address 127.0.98.1:37447)
0619 14:47:15.706391 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:15.706405 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:15.706505 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:15.707067 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:15.707070 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:15.707381 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:47:15.707480 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:15.707486 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:15.708287 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:15.708294 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:15.709073 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:15.709083 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:15.709794 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:15.709797 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:15.709884 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:15.718892 (+ 9008us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:47:15.718908 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:47:15.721720 (+ 2812us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43861: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":93,"threads_started":1}
W20260619 14:47:15.722189 9573 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:47:15.706244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37447 (local address 127.0.98.62:43861)
0619 14:47:15.706483 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:15.706487 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:15.706501 (+ 14us) server_negotiation.cc:407] Connection header received
0619 14:47:15.706801 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:15.706805 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:15.706857 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:15.706986 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:15.707615 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:15.708165 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:15.709242 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:15.709560 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:15.709683 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:15.722042 (+ 12359us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:37447: BlockingRecv error: recv got EOF from 127.0.98.1:37447 (error 108)
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":95,"threads_started":1}
Jun 19 14:47:15 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:47:15 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:47:15.913506 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:15.906601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43861 (local address 127.0.98.2:41135)
0619 14:47:15.907031 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:15.907053 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:15.907201 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:15.907586 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:15.907590 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:15.907840 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:47:15.907943 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:15.907949 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:15.909066 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:15.909069 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:15.909713 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:15.909723 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:15.910255 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:15.910258 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:15.910338 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:15.910858 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:47:15.910872 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:47:15.913373 (+ 2501us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43861: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":110,"threads_started":1}
W20260619 14:47:15.913936 9573 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:47:15.906760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41135 (local address 127.0.98.62:43861)
0619 14:47:15.906949 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:15.906956 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:15.907137 (+ 181us) server_negotiation.cc:407] Connection header received
0619 14:47:15.907295 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:15.907299 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:15.907357 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:15.907450 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:15.908108 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:15.908920 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:15.909899 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:15.910128 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:15.910305 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:15.913570 (+ 3265us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:41135: BlockingRecv error: recv got EOF from 127.0.98.2:41135 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Jun 19 14:47:16 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:47:16 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:47:16.121845 9575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:16.114315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43861 (local address 127.0.98.3:48175)
0619 14:47:16.114791 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:16.114813 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:16.114970 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:16.115280 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:16.115283 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:16.115484 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:47:16.115601 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:16.115606 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:16.117439 (+ 1833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:16.117442 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:16.118180 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:16.118192 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:16.118759 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:16.118763 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:16.118856 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:16.119482 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:47:16.119494 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:47:16.121656 (+ 2162us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43861: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":292,"threads_started":1}
W20260619 14:47:16.122107 9573 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:47:16.114486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48175 (local address 127.0.98.62:43861)
0619 14:47:16.114643 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:16.114647 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:16.114943 (+ 296us) server_negotiation.cc:407] Connection header received
0619 14:47:16.115063 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:16.115065 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:16.115115 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:16.115197 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:16.115892 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:16.117239 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:16.118390 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:16.118632 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:16.118754 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:16.121976 (+ 3222us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:48175: BlockingRecv error: recv got EOF from 127.0.98.3:48175 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Jun 19 14:47:16 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:47:16 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:47:16.730564 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:16.723130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43861 (local address 127.0.98.1:45023)
0619 14:47:16.723573 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:16.723588 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:16.723714 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:16.724169 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:16.724172 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:16.724385 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:47:16.724496 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:16.724502 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:16.725558 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:16.725562 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:16.726447 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:16.726459 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:16.727242 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:16.727247 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:16.727329 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:16.727944 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:47:16.727957 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:47:16.730382 (+ 2425us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43861: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":105,"threads_started":1}
W20260619 14:47:16.730674 9577 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:47:16.723215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45023 (local address 127.0.98.62:43861)
0619 14:47:16.723526 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:16.723529 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:16.723839 (+ 310us) server_negotiation.cc:407] Connection header received
0619 14:47:16.723893 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:16.723896 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:16.723945 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:16.724051 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:16.724709 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:16.725357 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:16.726589 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:16.727040 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:16.727166 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:16.730536 (+ 3370us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:45023: BlockingRecv error: recv got EOF from 127.0.98.1:45023 (error 108)
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":91,"threads_started":1}
Jun 19 14:47:16 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:47:16 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:47:16.921060 9578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:16.914610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43861 (local address 127.0.98.2:57499)
0619 14:47:16.914929 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:16.914947 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:16.915078 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:16.915491 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:16.915494 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:16.915723 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:47:16.915818 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:16.915824 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:16.916775 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:16.916780 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:16.917353 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:16.917361 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:16.917849 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:16.917853 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:16.917942 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:16.918554 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:47:16.918567 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:47:16.920934 (+ 2367us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43861: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":90,"threads_started":1}
W20260619 14:47:16.921366 9577 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:47:16.914687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57499 (local address 127.0.98.62:43861)
0619 14:47:16.914938 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:16.914942 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:16.915048 (+ 106us) server_negotiation.cc:407] Connection header received
0619 14:47:16.915195 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:16.915198 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:16.915254 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:16.915480 (+ 226us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:16.916039 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:16.916582 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:16.917486 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:16.917715 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:16.917834 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:16.921186 (+ 3352us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:57499: BlockingRecv error: recv got EOF from 127.0.98.2:57499 (error 108)
Metrics: {"server-negotiator.queue_time_us":144}
Jun 19 14:47:17 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:47:17 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:47:17.130239 9579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:17.122979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43861 (local address 127.0.98.3:54653)
0619 14:47:17.123430 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:17.123445 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:17.123569 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:17.123978 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:17.123981 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:17.124169 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:47:17.124276 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:17.124282 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:17.125179 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:17.125182 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:17.125893 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:17.125903 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:17.126484 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:17.126487 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:17.126563 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:17.127458 (+ 895us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:47:17.127474 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:47:17.130062 (+ 2588us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43861: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":248,"threads_started":1}
W20260619 14:47:17.130308 9577 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:47:17.123124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54653 (local address 127.0.98.62:43861)
0619 14:47:17.123269 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:17.123272 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:17.123572 (+ 300us) server_negotiation.cc:407] Connection header received
0619 14:47:17.123656 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:17.123658 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:17.123849 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:17.123924 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:17.124417 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:17.125044 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:17.126025 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:17.126335 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:17.126451 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:17.130200 (+ 3749us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:54653: BlockingRecv error: recv got EOF from 127.0.98.3:54653 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 19 14:47:17 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:47:17 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:47:17.738386 9580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:17.731858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43861 (local address 127.0.98.1:46013)
0619 14:47:17.732162 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:17.732176 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:17.732285 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:17.732592 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:17.732595 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:17.732825 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:47:17.732925 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:17.732931 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:17.734025 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:17.734028 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:17.734619 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:17.734628 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:17.735200 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:17.735204 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:17.735272 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:17.735843 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:47:17.735855 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:47:17.738202 (+ 2347us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43861: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":113,"threads_started":1}
W20260619 14:47:17.738566 9581 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:47:17.732027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46013 (local address 127.0.98.62:43861)
0619 14:47:17.732309 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:17.732320 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:17.732334 (+ 14us) server_negotiation.cc:407] Connection header received
0619 14:47:17.732376 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:17.732378 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:17.732421 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:17.732515 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:17.733078 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:17.733658 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:17.734826 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:17.735052 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:17.735149 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:17.738415 (+ 3266us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:46013: BlockingRecv error: recv got EOF from 127.0.98.1:46013 (error 108)
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":79,"threads_started":1}
Jun 19 14:47:17 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:47:17 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:47:17.928804 9582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:17.922041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43861 (local address 127.0.98.2:46515)
0619 14:47:17.922427 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:17.922445 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:17.922573 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:17.922949 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:17.922955 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:17.923153 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:47:17.923252 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:17.923259 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:17.924202 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:17.924208 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:17.924802 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:17.924811 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:17.925304 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:17.925307 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:17.925387 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:17.926407 (+ 1020us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:47:17.926420 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:47:17.928619 (+ 2199us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43861: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":154,"threads_started":1}
W20260619 14:47:17.929023 9581 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:47:17.922305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46515 (local address 127.0.98.62:43861)
0619 14:47:17.922482 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:17.922489 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:17.922506 (+ 17us) server_negotiation.cc:407] Connection header received
0619 14:47:17.922689 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:17.922692 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:17.922748 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:17.922833 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:17.923392 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:17.924071 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:17.924936 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:17.925167 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:17.925296 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:17.928910 (+ 3614us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:46515: BlockingRecv error: recv got EOF from 127.0.98.2:46515 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Jun 19 14:47:18 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:47:18 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:47:18.138183 9583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:18.131293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43861 (local address 127.0.98.3:54157)
0619 14:47:18.131605 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:18.131625 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:18.131751 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:18.132128 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:18.132132 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:18.132360 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:47:18.132463 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:18.132469 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:18.133396 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:18.133402 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:18.134055 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:18.134066 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:18.134675 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:18.134679 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:18.134776 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:18.135535 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:47:18.135549 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:47:18.138021 (+ 2472us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43861: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":125,"threads_started":1}
W20260619 14:47:18.138525 9581 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:47:18.131615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54157 (local address 127.0.98.62:43861)
0619 14:47:18.131845 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:18.131852 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:18.131869 (+ 17us) server_negotiation.cc:407] Connection header received
0619 14:47:18.131942 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:18.131944 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:18.131992 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:18.132099 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:18.132614 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:18.133260 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:18.134213 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:18.134482 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:18.134667 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:18.138401 (+ 3734us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:54157: BlockingRecv error: recv got EOF from 127.0.98.3:54157 (error 108)
Metrics: {"server-negotiator.queue_time_us":108}
Jun 19 14:47:18 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:47:18 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:47:18.746439 9584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:18.739640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43861 (local address 127.0.98.1:45035)
0619 14:47:18.739957 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:18.739976 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:18.740124 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:18.740707 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:18.740710 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:18.740947 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:47:18.741074 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:18.741082 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:18.742169 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:18.742172 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:18.742777 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:18.742788 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:18.743347 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:18.743350 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:18.743428 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:18.743953 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:47:18.743965 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:47:18.746305 (+ 2340us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43861: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":85,"threads_started":1}
W20260619 14:47:18.746685 9585 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:47:18.739906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45035 (local address 127.0.98.62:43861)
0619 14:47:18.740218 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:18.740222 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:18.740238 (+ 16us) server_negotiation.cc:407] Connection header received
0619 14:47:18.740278 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:18.740281 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:18.740333 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:18.740438 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:18.741220 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:18.742022 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:18.742923 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:18.743156 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:18.743251 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:18.746523 (+ 3272us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:45035: BlockingRecv error: recv got EOF from 127.0.98.1:45035 (error 108)
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":90,"threads_started":1}
Jun 19 14:47:18 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:47:18 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:47:18.936625 9586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:18.930011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43861 (local address 127.0.98.2:43827)
0619 14:47:18.930410 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:18.930424 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:18.930589 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:18.930921 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:18.930924 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:18.931147 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:47:18.931243 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:18.931249 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:18.932125 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:18.932131 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:18.932938 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:18.932950 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:18.933487 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:18.933492 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:18.933585 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:18.934252 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:47:18.934272 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:47:18.936402 (+ 2130us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43861: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":74,"threads_started":1}
W20260619 14:47:18.936750 9585 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:47:18.930050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43827 (local address 127.0.98.62:43861)
0619 14:47:18.930263 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:18.930269 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:18.930528 (+ 259us) server_negotiation.cc:407] Connection header received
0619 14:47:18.930713 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:18.930716 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:18.930784 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:18.930893 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:18.931402 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:18.931989 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:18.933083 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:18.933347 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:18.933478 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:18.936567 (+ 3089us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:43827: BlockingRecv error: recv got EOF from 127.0.98.2:43827 (error 108)
Metrics: {"server-negotiator.queue_time_us":113}
Jun 19 14:47:19 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:47:19 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:47:19.147086 9587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:19.140070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43861 (local address 127.0.98.3:33023)
0619 14:47:19.140525 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:19.140540 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:19.140851 (+ 311us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:19.141241 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:19.141244 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:19.141597 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:47:19.141709 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:19.141715 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:19.142725 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:19.142729 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:19.143310 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:19.143318 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:19.143827 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:19.143830 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:19.143906 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:19.144596 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:47:19.144609 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:47:19.146962 (+ 2353us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43861: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":107,"threads_started":1}
W20260619 14:47:19.147471 9585 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:47:19.140179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33023 (local address 127.0.98.62:43861)
0619 14:47:19.140410 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:19.140421 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:19.140626 (+ 205us) server_negotiation.cc:407] Connection header received
0619 14:47:19.141031 (+ 405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:19.141034 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:19.141092 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:19.141180 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:19.141949 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:19.142589 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:19.143450 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:19.143683 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:19.143782 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:19.147161 (+ 3379us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:33023: BlockingRecv error: recv got EOF from 127.0.98.3:33023 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Jun 19 14:47:19 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:47:19 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:47:19.754882 9588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:19.747916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43861 (local address 127.0.98.1:60745)
0619 14:47:19.748188 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:19.748211 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:19.748361 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:19.748781 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:19.748785 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:19.749071 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:47:19.749218 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:19.749226 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:19.750386 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:19.750389 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:19.751293 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:19.751304 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:19.751852 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:19.751855 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:19.751964 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:19.752692 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:47:19.752704 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:47:19.754754 (+ 2050us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43861: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":106,"threads_started":1}
W20260619 14:47:19.755218 9589 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:47:19.747959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60745 (local address 127.0.98.62:43861)
0619 14:47:19.748256 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:19.748264 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:19.748311 (+ 47us) server_negotiation.cc:407] Connection header received
0619 14:47:19.748510 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:19.748515 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:19.748574 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:19.748687 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:19.749394 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:19.750226 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:19.751495 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:19.751725 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:19.751856 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:19.755064 (+ 3208us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:60745: BlockingRecv error: recv got EOF from 127.0.98.1:60745 (error 108)
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":78,"threads_started":1}
Jun 19 14:47:19 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:47:19 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:47:19.945156 9590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:19.937985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43861 (local address 127.0.98.2:49893)
0619 14:47:19.938392 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:19.938407 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:19.938605 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:19.939070 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:19.939073 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:19.939262 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:47:19.939370 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:19.939376 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:19.940302 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:19.940305 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:19.941148 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:19.941159 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:19.941767 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:19.941771 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:19.941844 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:19.942444 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:47:19.942457 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:47:19.945033 (+ 2576us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43861: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":152,"threads_started":1}
W20260619 14:47:19.945580 9589 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:47:19.938129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49893 (local address 127.0.98.62:43861)
0619 14:47:19.938426 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:19.938433 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:19.938596 (+ 163us) server_negotiation.cc:407] Connection header received
0619 14:47:19.938706 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:19.938708 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:19.938755 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:19.938839 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:19.939550 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:19.940162 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:19.941354 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:19.941603 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:19.941719 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:19.945344 (+ 3625us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:49893: BlockingRecv error: recv got EOF from 127.0.98.2:49893 (error 108)
Metrics: {"server-negotiator.queue_time_us":182}
Jun 19 14:47:20 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:47:20 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:47:20.154642 9591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:20.148325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43861 (local address 127.0.98.3:45939)
0619 14:47:20.148685 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:20.148702 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:20.148811 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:20.149092 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:20.149095 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:20.149303 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:47:20.149506 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:20.149514 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:20.150433 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:20.150440 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:20.151005 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:20.151013 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:20.151514 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:20.151517 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:20.151603 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:20.152161 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:47:20.152173 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:47:20.154482 (+ 2309us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43861: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":92,"threads_started":1}
W20260619 14:47:20.154860 9589 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:47:20.148586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45939 (local address 127.0.98.62:43861)
0619 14:47:20.148763 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:20.148769 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:20.148783 (+ 14us) server_negotiation.cc:407] Connection header received
0619 14:47:20.148928 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:20.148931 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:20.148984 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:20.149073 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:20.149660 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:20.150301 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:20.151152 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:20.151389 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:20.151507 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:20.154722 (+ 3215us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:45939: BlockingRecv error: recv got EOF from 127.0.98.3:45939 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Jun 19 14:47:20 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:47:20 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:47:20.766129 9592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:20.755789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43861 (local address 127.0.98.1:47383)
0619 14:47:20.756175 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:20.756189 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:20.756287 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:20.756673 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:20.756675 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:20.756978 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:47:20.757082 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:20.757088 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:20.758029 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:20.758031 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:20.758620 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:20.758630 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:20.759156 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:20.759159 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:20.759221 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:20.763595 (+ 4374us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:47:20.763611 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:47:20.766002 (+ 2391us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43861: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":81,"threads_started":1}
W20260619 14:47:20.766438 9593 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:47:20.756034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47383 (local address 127.0.98.62:43861)
0619 14:47:20.756292 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:20.756296 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:20.756311 (+ 15us) server_negotiation.cc:407] Connection header received
0619 14:47:20.756434 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:20.756437 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:20.756494 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:20.756598 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:20.757229 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:20.757774 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:20.758770 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:20.759030 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:20.759132 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:20.766290 (+ 7158us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:47383: BlockingRecv error: recv got EOF from 127.0.98.1:47383 (error 108)
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":95,"threads_started":1}
Jun 19 14:47:20 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:47:20 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:47:20.953959 9594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:20.946156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43861 (local address 127.0.98.2:53967)
0619 14:47:20.946494 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:20.946516 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:20.946633 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:20.947067 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:20.947070 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:20.947262 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:47:20.947380 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:20.947386 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:20.948304 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:20.948309 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:20.949445 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:20.949458 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:20.950355 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:20.950360 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:20.950442 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:20.951353 (+ 911us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:47:20.951365 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:47:20.953806 (+ 2441us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43861: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":72,"threads_started":1}
W20260619 14:47:20.954353 9593 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:47:20.946426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53967 (local address 127.0.98.62:43861)
0619 14:47:20.946590 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:20.946594 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:20.946611 (+ 17us) server_negotiation.cc:407] Connection header received
0619 14:47:20.946753 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:20.946756 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:20.946813 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:20.947009 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:20.947538 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:20.948122 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:20.949607 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:20.950095 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:20.950243 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:20.954260 (+ 4017us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:53967: BlockingRecv error: recv got EOF from 127.0.98.2:53967 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Jun 19 14:47:21 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:47:21 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:47:21.162544 9595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:21.155683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43861 (local address 127.0.98.3:54791)
0619 14:47:21.156176 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:21.156192 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:21.156387 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:21.156721 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:21.156723 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:21.156959 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:47:21.157066 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:21.157071 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:21.157929 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:21.157933 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:21.158671 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:21.158681 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:21.159221 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:21.159225 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:21.159316 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:21.160150 (+ 834us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:47:21.160163 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:47:21.162364 (+ 2201us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43861: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":270,"threads_started":1}
W20260619 14:47:21.162603 9593 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:47:21.155925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54791 (local address 127.0.98.62:43861)
0619 14:47:21.156171 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:21.156175 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:21.156284 (+ 109us) server_negotiation.cc:407] Connection header received
0619 14:47:21.156508 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:21.156511 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:21.156560 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:21.156666 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:21.157197 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:21.157777 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:21.158817 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:21.159050 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:21.159140 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:21.162495 (+ 3355us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:54791: BlockingRecv error: recv got EOF from 127.0.98.3:54791 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
Jun 19 14:47:21 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:47:21 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:47:21.774314 9596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:21.767430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43861 (local address 127.0.98.1:52387)
0619 14:47:21.767758 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:21.767775 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:21.767894 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:21.768202 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:21.768204 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:21.768411 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:47:21.768506 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:21.768512 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:21.769611 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:21.769614 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:21.770188 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:21.770197 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:21.770827 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:21.770831 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:21.771033 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:21.771626 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:47:21.771638 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:47:21.774124 (+ 2486us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43861: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":111,"threads_started":1}
W20260619 14:47:21.774478 9597 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:47:21.767624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52387 (local address 127.0.98.62:43861)
0619 14:47:21.767974 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:21.767978 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:21.767992 (+ 14us) server_negotiation.cc:407] Connection header received
0619 14:47:21.768033 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:21.768035 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:21.768077 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:21.768164 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:21.768688 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:21.769332 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:21.770408 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:21.770662 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:21.770771 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:21.774261 (+ 3490us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:52387: BlockingRecv error: recv got EOF from 127.0.98.1:52387 (error 108)
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":80,"threads_started":1}
Jun 19 14:47:21 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:47:21 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:47:21.960932 9598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:21.955017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43861 (local address 127.0.98.2:37559)
0619 14:47:21.955324 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:21.955342 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:21.955469 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:21.955790 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:21.955793 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:21.956007 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:47:21.956098 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:21.956104 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:21.956981 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:21.956986 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:21.957554 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:21.957562 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:21.958028 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:21.958032 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:21.958116 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:21.958713 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:47:21.958725 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:47:21.960743 (+ 2018us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43861: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":125,"threads_started":1}
W20260619 14:47:21.960974 9597 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:47:21.955219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37559 (local address 127.0.98.62:43861)
0619 14:47:21.955449 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:21.955454 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:21.955471 (+ 17us) server_negotiation.cc:407] Connection header received
0619 14:47:21.955588 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:21.955591 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:21.955666 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:21.955765 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:21.956224 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:21.956871 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:21.957680 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:21.957914 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:21.958025 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:21.960872 (+ 2847us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:37559: BlockingRecv error: recv got EOF from 127.0.98.2:37559 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
Jun 19 14:47:22 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:47:22 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:47:22.170804 9599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:22.163789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43861 (local address 127.0.98.3:44161)
0619 14:47:22.164176 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:22.164189 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:22.164337 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:22.164630 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:22.164633 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:22.164874 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:47:22.164980 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:22.164986 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:22.165895 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:22.165899 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:22.166652 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:22.166665 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:22.167442 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:22.167446 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:22.167558 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:22.168299 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:47:22.168314 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:47:22.170680 (+ 2366us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43861: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":103,"threads_started":1}
W20260619 14:47:22.171221 9597 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:47:22.163913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44161 (local address 127.0.98.62:43861)
0619 14:47:22.164100 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:22.164104 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:22.164286 (+ 182us) server_negotiation.cc:407] Connection header received
0619 14:47:22.164458 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:22.164461 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:22.164514 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:22.164610 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:22.165117 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:22.165727 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:22.166975 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:22.167278 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:22.167472 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:22.171125 (+ 3653us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:44161: BlockingRecv error: recv got EOF from 127.0.98.3:44161 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
Jun 19 14:47:22 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:47:22 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:47:22.782231 9600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:22.775453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43861 (local address 127.0.98.1:56721)
0619 14:47:22.775726 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:22.775741 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:22.775851 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:22.776255 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:22.776258 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:22.776463 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:47:22.776565 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:22.776571 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:22.777590 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:22.777593 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:22.778207 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:22.778217 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:22.778693 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:22.778696 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:22.778761 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:22.779472 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:47:22.779484 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:47:22.782080 (+ 2596us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43861: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":96,"threads_started":1}
W20260619 14:47:22.782538 9601 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:47:22.775519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56721 (local address 127.0.98.62:43861)
0619 14:47:22.775797 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:22.775801 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:22.775818 (+ 17us) server_negotiation.cc:407] Connection header received
0619 14:47:22.776011 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:22.776015 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:22.776063 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:22.776180 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:22.776755 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:22.777440 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:22.778329 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:22.778576 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:22.778843 (+ 267us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:22.782341 (+ 3498us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:56721: BlockingRecv error: recv got EOF from 127.0.98.1:56721 (error 108)
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":120,"threads_started":1}
Jun 19 14:47:22 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:47:22 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:47:22.968621 9602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:22.961850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43861 (local address 127.0.98.2:60423)
0619 14:47:22.962213 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:22.962230 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:22.962381 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:22.962687 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:22.962690 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:22.962906 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:47:22.963001 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:22.963006 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:22.963794 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:22.963800 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:22.964382 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:22.964391 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:22.965262 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:22.965266 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:22.965364 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:22.966023 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:47:22.966037 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:47:22.968479 (+ 2442us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43861: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":131,"threads_started":1}
W20260619 14:47:22.968986 9601 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:47:22.962054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60423 (local address 127.0.98.62:43861)
0619 14:47:22.962215 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:22.962220 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:22.962318 (+ 98us) server_negotiation.cc:407] Connection header received
0619 14:47:22.962485 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:22.962488 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:22.962546 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:22.962656 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:22.963137 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:22.963661 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:22.964538 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:22.965064 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:22.965248 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:22.968867 (+ 3619us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:60423: BlockingRecv error: recv got EOF from 127.0.98.2:60423 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Jun 19 14:47:23 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:47:23 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:47:23.178996 9603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:23.172007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43861 (local address 127.0.98.3:47021)
0619 14:47:23.172402 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:23.172422 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:23.172539 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:23.172967 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:23.172970 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:23.173168 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:47:23.173268 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:23.173273 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:23.174207 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:23.174211 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:23.174925 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:23.174937 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:23.175521 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:23.175525 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:23.175593 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:23.176360 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:47:23.176372 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:47:23.178849 (+ 2477us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43861: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":98,"threads_started":1}
W20260619 14:47:23.179239 9601 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:47:23.172258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47021 (local address 127.0.98.62:43861)
0619 14:47:23.172485 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:23.172489 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:23.172505 (+ 16us) server_negotiation.cc:407] Connection header received
0619 14:47:23.172638 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:23.172663 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:23.172723 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:23.172803 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:23.173434 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:23.174064 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:23.175086 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:23.175364 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:23.175471 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:23.179113 (+ 3642us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:47021: BlockingRecv error: recv got EOF from 127.0.98.3:47021 (error 108)
Metrics: {"server-negotiator.queue_time_us":133}
Jun 19 14:47:23 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:47:23 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:47:23.790182 9604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:23.783108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43861 (local address 127.0.98.1:46529)
0619 14:47:23.783581 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:23.783597 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:23.783714 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:23.784080 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:23.784083 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:23.784273 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:47:23.784412 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:23.784419 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:23.785555 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:23.785558 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:23.786136 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:23.786145 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:23.786723 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:23.786726 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:23.786936 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:23.787571 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:47:23.787584 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:47:23.790034 (+ 2450us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43861: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":87,"threads_started":1}
W20260619 14:47:23.790550 9605 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:47:23.783273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46529 (local address 127.0.98.62:43861)
0619 14:47:23.783580 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:23.783591 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:23.783670 (+ 79us) server_negotiation.cc:407] Connection header received
0619 14:47:23.783858 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:23.783862 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:23.783912 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:23.784007 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:23.784573 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:23.785352 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:23.786309 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:23.786579 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:23.786682 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:23.790226 (+ 3544us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:46529: BlockingRecv error: recv got EOF from 127.0.98.1:46529 (error 108)
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":63,"threads_started":1}
Jun 19 14:47:23 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:47:23 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:47:23.977596 9606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:23.970105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43861 (local address 127.0.98.2:59393)
0619 14:47:23.970381 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:23.970398 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:23.970581 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:23.971200 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:23.971207 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:23.971579 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:47:23.971729 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:23.971738 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:23.972851 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:23.972854 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:23.973509 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:23.973519 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:23.974111 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:23.974114 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:23.974186 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:23.974836 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:47:23.974942 (+ 106us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:47:23.977433 (+ 2491us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43861: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":105,"threads_started":1}
W20260619 14:47:23.977792 9605 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:47:23.970171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59393 (local address 127.0.98.62:43861)
0619 14:47:23.970328 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:23.970333 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:23.970512 (+ 179us) server_negotiation.cc:407] Connection header received
0619 14:47:23.970760 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:23.970767 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:23.970824 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:23.971057 (+ 233us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:23.971912 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:23.972714 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:23.973676 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:23.973922 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:23.974050 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:23.977647 (+ 3597us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:59393: BlockingRecv error: recv got EOF from 127.0.98.2:59393 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Jun 19 14:47:24 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:47:24 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:47:24.187083 9607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:24.180041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43861 (local address 127.0.98.3:41435)
0619 14:47:24.180349 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:24.180367 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:24.180474 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:24.180870 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:24.180873 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:24.181056 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:47:24.181155 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:24.181161 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:24.182193 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:24.182196 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:24.182924 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:24.182936 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:24.183687 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:24.183692 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:24.183768 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:24.184699 (+ 931us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:47:24.184713 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:47:24.186878 (+ 2165us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43861: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":113,"threads_started":1}
W20260619 14:47:24.187271 9605 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:47:24.180204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41435 (local address 127.0.98.62:43861)
0619 14:47:24.180346 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:24.180350 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:24.180471 (+ 121us) server_negotiation.cc:407] Connection header received
0619 14:47:24.180547 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:24.180550 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:24.180603 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:24.180768 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:24.181286 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:24.181998 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:24.183097 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:24.183357 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:24.183478 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:24.187147 (+ 3669us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:41435: BlockingRecv error: recv got EOF from 127.0.98.3:41435 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 19 14:47:24 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:47:24 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:47:24.798431 9608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:24.791375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43861 (local address 127.0.98.1:59231)
0619 14:47:24.791755 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:24.791771 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:24.791908 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:24.792305 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:24.792308 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:24.792494 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:47:24.792589 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:24.792595 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:24.793560 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:24.793563 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:24.794192 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:24.794203 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:24.794817 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:24.794820 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:24.794890 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:24.795558 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:47:24.795576 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:47:24.798273 (+ 2697us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43861: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":116,"threads_started":1}
W20260619 14:47:24.798621 9609 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:47:24.791555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59231 (local address 127.0.98.62:43861)
0619 14:47:24.791785 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:24.791789 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:24.791845 (+ 56us) server_negotiation.cc:407] Connection header received
0619 14:47:24.792092 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:24.792095 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:24.792144 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:24.792233 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:24.792790 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:24.793389 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:24.794324 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:24.794647 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:24.794812 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:24.798473 (+ 3661us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:59231: BlockingRecv error: recv got EOF from 127.0.98.1:59231 (error 108)
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":59,"threads_started":1}
Jun 19 14:47:24 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 19 14:47:24 dist-test-slave-2sb1 krb5kdc[8551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260619 14:47:24.985348 9610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:24.978781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43861 (local address 127.0.98.2:57561)
0619 14:47:24.979049 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:24.979072 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:24.979172 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:24.979469 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:24.979536 (+ 67us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:24.979735 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:47:24.979909 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:24.979916 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:24.980852 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:24.980859 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:24.981605 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:24.981616 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:24.982229 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:24.982232 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:24.982307 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:24.982874 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:47:24.982891 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:47:24.985194 (+ 2303us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:43861: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":86,"threads_started":1}
W20260619 14:47:24.985553 9609 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:47:24.978956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57561 (local address 127.0.98.62:43861)
0619 14:47:24.979107 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:24.979110 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:24.979150 (+ 40us) server_negotiation.cc:407] Connection header received
0619 14:47:24.979267 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:24.979270 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:24.979313 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:24.979407 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:24.980028 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:24.980737 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:24.981739 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:24.982081 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:24.982171 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:24.985400 (+ 3229us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:57561: BlockingRecv error: recv got EOF from 127.0.98.2:57561 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 14:47:25.055084 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 9134
I20260619 14:47:25.062582 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 9286
I20260619 14:47:25.068958 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 9418
I20260619 14:47:25.074842 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 9064
2026-06-19T14:47:25Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18272 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-19T14:47:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:47:25Z Disabled control of system clock
I20260619 14:47:25.107509 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880383132139-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880383132139-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880383132139-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45705
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:39137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45705
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260619 14:47:25.231148 9614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:47:25.231478 9614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:47:25.231516 9614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:47:25.236981 9614 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:47:25.237076 9614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:47:25.237105 9614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:47:25.237143 9614 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:47:25.237178 9614 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:47:25.247318 9614 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880383132139-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880383132139-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45705
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45705
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9614
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880383132139-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:47:25.249114 9614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:47:25.251585 9614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 14:47:25.263552 9614 server_base.cc:1034] running on GCE node
W20260619 14:47:25.263579 9619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:47:25.263625 9622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:47:25.263803 9620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:47:25.264810 9614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:47:25.266116 9614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:47:25.267341 9614 hybrid_clock.cc:648] HybridClock initialized: now 1781880445267300 us; error 70 us; skew 500 ppm
I20260619 14:47:25.270329 9614 webserver.cc:460] Webserver started at http://127.0.98.62:42365/ using document root <none> and password file <none>
I20260619 14:47:25.271010 9614 fs_manager.cc:359] Metadata directory not provided
I20260619 14:47:25.271119 9614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:47:25.271356 9614 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:47:25.273886 9614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880383132139-392-0/minicluster-data/master-0/data/instance:
uuid: "950cb46c1f5c43279862264c9413434f"
format_stamp: "Formatted at 2026-06-19 14:47:25 on dist-test-slave-2sb1"
server_key: "5739b1a3cd7bb76febb80568e9c10ac7"
server_key_iv: "3e626e35f579f826ff57a18217f155c2"
server_key_version: "encryptionkey@0"
I20260619 14:47:25.275015 9614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880383132139-392-0/minicluster-data/master-0/wal/instance:
uuid: "950cb46c1f5c43279862264c9413434f"
format_stamp: "Formatted at 2026-06-19 14:47:25 on dist-test-slave-2sb1"
server_key: "5739b1a3cd7bb76febb80568e9c10ac7"
server_key_iv: "3e626e35f579f826ff57a18217f155c2"
server_key_version: "encryptionkey@0"
I20260619 14:47:25.280324 9614 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260619 14:47:25.285346 9628 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:47:25.287631 9614 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260619 14:47:25.287827 9614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880383132139-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880383132139-392-0/minicluster-data/master-0/wal
uuid: "950cb46c1f5c43279862264c9413434f"
format_stamp: "Formatted at 2026-06-19 14:47:25 on dist-test-slave-2sb1"
server_key: "5739b1a3cd7bb76febb80568e9c10ac7"
server_key_iv: "3e626e35f579f826ff57a18217f155c2"
server_key_version: "encryptionkey@0"
I20260619 14:47:25.287972 9614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880383132139-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880383132139-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880383132139-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:47:25.301709 9614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:47:25.302532 9614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:47:25.302752 9614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:47:25.310316 9614 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45705
I20260619 14:47:25.310371 9680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45705 every 8 connection(s)
I20260619 14:47:25.311374 9614 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:47:25.314918 9681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:47:25.315189 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 9614
I20260619 14:47:25.315315 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880383132139-392-0/minicluster-data/master-0/wal/instance
I20260619 14:47:25.315680 392 external_mini_cluster.cc:1428] Setting key 7d139b89e7519d45c1922f42c3eb20ed
I20260619 14:47:25.323136 9681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 950cb46c1f5c43279862264c9413434f: Bootstrap starting.
I20260619 14:47:25.326237 9681 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 950cb46c1f5c43279862264c9413434f: Neither blocks nor log segments found. Creating new log.
I20260619 14:47:25.326433 9684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:25.317163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33424 (local address 127.0.98.62:45705)
0619 14:47:25.317862 (+ 699us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:25.317878 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:25.317917 (+ 39us) server_negotiation.cc:407] Connection header received
0619 14:47:25.318851 (+ 934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:25.318878 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:25.319357 (+ 479us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:25.319787 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:25.320302 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:25.321417 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:25.322359 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:25.322868 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:25.323059 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:25.324063 (+ 1004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:47:25.324068 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:47:25.324082 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:47:25.324523 (+ 441us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:47:25.324585 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:47:25.324608 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:47:25.324828 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:25.325155 (+ 327us) server_negotiation.cc:299] Negotiation successful
0619 14:47:25.325454 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":126,"threads_started":1}
I20260619 14:47:25.327308 9681 log.cc:824] T 00000000000000000000000000000000 P 950cb46c1f5c43279862264c9413434f: Log is configured to *not* fsync() on all Append() calls
I20260619 14:47:25.330138 9681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 950cb46c1f5c43279862264c9413434f: No bootstrap required, opened a new log
I20260619 14:47:25.334842 9681 raft_consensus.cc:348] T 00000000000000000000000000000000 P 950cb46c1f5c43279862264c9413434f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "950cb46c1f5c43279862264c9413434f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45705 } }
I20260619 14:47:25.335145 9681 raft_consensus.cc:374] T 00000000000000000000000000000000 P 950cb46c1f5c43279862264c9413434f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:47:25.335193 9681 raft_consensus.cc:729] T 00000000000000000000000000000000 P 950cb46c1f5c43279862264c9413434f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 950cb46c1f5c43279862264c9413434f, State: Initialized, Role: FOLLOWER
I20260619 14:47:25.335762 9681 consensus_queue.cc:260] T 00000000000000000000000000000000 P 950cb46c1f5c43279862264c9413434f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "950cb46c1f5c43279862264c9413434f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45705 } }
I20260619 14:47:25.335992 9681 raft_consensus.cc:388] T 00000000000000000000000000000000 P 950cb46c1f5c43279862264c9413434f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:47:25.336071 9681 raft_consensus.cc:482] T 00000000000000000000000000000000 P 950cb46c1f5c43279862264c9413434f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:47:25.336179 9681 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 950cb46c1f5c43279862264c9413434f [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:47:25.337519 9681 raft_consensus.cc:504] T 00000000000000000000000000000000 P 950cb46c1f5c43279862264c9413434f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "950cb46c1f5c43279862264c9413434f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45705 } }
I20260619 14:47:25.337934 9681 leader_election.cc:302] T 00000000000000000000000000000000 P 950cb46c1f5c43279862264c9413434f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 950cb46c1f5c43279862264c9413434f; no voters:
I20260619 14:47:25.338454 9681 leader_election.cc:288] T 00000000000000000000000000000000 P 950cb46c1f5c43279862264c9413434f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:47:25.338551 9686 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 950cb46c1f5c43279862264c9413434f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:47:25.338752 9686 raft_consensus.cc:686] T 00000000000000000000000000000000 P 950cb46c1f5c43279862264c9413434f [term 1 LEADER]: Becoming Leader. State: Replica: 950cb46c1f5c43279862264c9413434f, State: Running, Role: LEADER
I20260619 14:47:25.339080 9686 consensus_queue.cc:237] T 00000000000000000000000000000000 P 950cb46c1f5c43279862264c9413434f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "950cb46c1f5c43279862264c9413434f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45705 } }
I20260619 14:47:25.339879 9681 sys_catalog.cc:564] T 00000000000000000000000000000000 P 950cb46c1f5c43279862264c9413434f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:47:25.340921 9687 sys_catalog.cc:455] T 00000000000000000000000000000000 P 950cb46c1f5c43279862264c9413434f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "950cb46c1f5c43279862264c9413434f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "950cb46c1f5c43279862264c9413434f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45705 } } }
I20260619 14:47:25.340926 9688 sys_catalog.cc:455] T 00000000000000000000000000000000 P 950cb46c1f5c43279862264c9413434f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 950cb46c1f5c43279862264c9413434f. Latest consensus state: current_term: 1 leader_uuid: "950cb46c1f5c43279862264c9413434f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "950cb46c1f5c43279862264c9413434f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45705 } } }
I20260619 14:47:25.341109 9688 sys_catalog.cc:458] T 00000000000000000000000000000000 P 950cb46c1f5c43279862264c9413434f [sys.catalog]: This master's current role is: LEADER
I20260619 14:47:25.341107 9687 sys_catalog.cc:458] T 00000000000000000000000000000000 P 950cb46c1f5c43279862264c9413434f [sys.catalog]: This master's current role is: LEADER
I20260619 14:47:25.341516 9695 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:47:25.345299 9695 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:47:25.352020 9695 catalog_manager.cc:1347] Generated new cluster ID: fc0253f09c8c4fca83b85dba37ed0a69
I20260619 14:47:25.352339 9695 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:47:25.385800 9695 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:47:25.386880 9695 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:47:25.400372 9695 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 950cb46c1f5c43279862264c9413434f: Generated new TSK 0
I20260619 14:47:25.401254 9695 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:47:25.411010 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880383132139-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880383132139-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880383132139-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880383132139-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45705
--builtin_ntp_servers=127.0.98.20:39137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260619 14:47:25.549569 9705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:47:25.549891 9705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:47:25.549933 9705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:47:25.554405 9705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:47:25.554482 9705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:47:25.554543 9705 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 14:47:25.560021 9705 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880383132139-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880383132139-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880383132139-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45705
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9705
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880383132139-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:47:25.561329 9705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:47:25.562810 9705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:47:25.573563 9710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:47:25.573681 9711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:47:25.573580 9713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:47:25.574097 9705 server_base.cc:1034] running on GCE node
I20260619 14:47:25.574769 9705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:47:25.575618 9705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:47:25.576789 9705 hybrid_clock.cc:648] HybridClock initialized: now 1781880445576750 us; error 61 us; skew 500 ppm
I20260619 14:47:25.579350 9705 webserver.cc:460] Webserver started at http://127.0.98.1:41947/ using document root <none> and password file <none>
I20260619 14:47:25.580250 9705 fs_manager.cc:359] Metadata directory not provided
I20260619 14:47:25.580354 9705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:47:25.580701 9705 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:47:25.583572 9705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880383132139-392-0/minicluster-data/ts-0/data/instance:
uuid: "67ebec6b95b24d72bc9485ff946c58a9"
format_stamp: "Formatted at 2026-06-19 14:47:25 on dist-test-slave-2sb1"
server_key: "e7917ab908435fed8a8ff6e0e645352b"
server_key_iv: "569e1050dc4dc222fcf93cffb61ca8b6"
server_key_version: "encryptionkey@0"
I20260619 14:47:25.584273 9705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880383132139-392-0/minicluster-data/ts-0/wal/instance:
uuid: "67ebec6b95b24d72bc9485ff946c58a9"
format_stamp: "Formatted at 2026-06-19 14:47:25 on dist-test-slave-2sb1"
server_key: "e7917ab908435fed8a8ff6e0e645352b"
server_key_iv: "569e1050dc4dc222fcf93cffb61ca8b6"
server_key_version: "encryptionkey@0"
I20260619 14:47:25.589088 9705 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 14:47:25.592200 9719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:47:25.594028 9705 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:47:25.594180 9705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880383132139-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880383132139-392-0/minicluster-data/ts-0/wal
uuid: "67ebec6b95b24d72bc9485ff946c58a9"
format_stamp: "Formatted at 2026-06-19 14:47:25 on dist-test-slave-2sb1"
server_key: "e7917ab908435fed8a8ff6e0e645352b"
server_key_iv: "569e1050dc4dc222fcf93cffb61ca8b6"
server_key_version: "encryptionkey@0"
I20260619 14:47:25.594370 9705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880383132139-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880383132139-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880383132139-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:47:25.638002 9705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:47:25.638836 9705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:47:25.639083 9705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:47:25.639772 9705 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:47:25.641184 9705 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:47:25.641264 9705 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:47:25.641338 9705 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:47:25.641369 9705 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:47:25.652266 9705 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40095
I20260619 14:47:25.652283 9832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40095 every 8 connection(s)
I20260619 14:47:25.653421 9705 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880383132139-392-0/minicluster-data/ts-0/data/info.pb
I20260619 14:47:25.660620 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 9705
I20260619 14:47:25.660806 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880383132139-392-0/minicluster-data/ts-0/wal/instance
I20260619 14:47:25.661159 392 external_mini_cluster.cc:1428] Setting key cdbb5093226975c7a0a5dccacc6f1f01
I20260619 14:47:25.663424 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880383132139-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880383132139-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880383132139-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880383132139-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45705
--builtin_ntp_servers=127.0.98.20:39137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260619 14:47:25.665409 9684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:25.655701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33347 (local address 127.0.98.62:45705)
0619 14:47:25.655916 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:25.655922 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:25.657658 (+ 1736us) server_negotiation.cc:407] Connection header received
0619 14:47:25.659125 (+ 1467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:25.659129 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:25.659180 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:25.659288 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:25.660365 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:25.661245 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:25.662818 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:25.663045 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:25.663134 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:25.664557 (+ 1423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:47:25.664560 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:47:25.664563 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:47:25.664576 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:47:25.664597 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:47:25.664606 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:47:25.664842 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:25.665162 (+ 320us) server_negotiation.cc:299] Negotiation successful
0619 14:47:25.665240 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260619 14:47:25.666520 9835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:25.656509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45705 (local address 127.0.98.1:33347)
0619 14:47:25.657464 (+ 955us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:25.657513 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:25.658865 (+ 1352us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:25.659535 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:25.659542 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:25.659644 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:47:25.660170 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:25.660182 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:25.661423 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:25.661427 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:25.662661 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:25.662672 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:25.663229 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:25.663236 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:25.663495 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:25.664174 (+ 679us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:47:25.664179 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:47:25.664240 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:47:25.664344 (+ 104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:47:25.664834 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:47:25.664846 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:47:25.664855 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 14:47:25.665142 (+ 287us) client_negotiation.cc:240] Negotiation successful
0619 14:47:25.665501 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":219,"threads_started":1}
I20260619 14:47:25.668056 9833 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45705
I20260619 14:47:25.668373 9833 heartbeater.cc:464] Registering TS with master...
I20260619 14:47:25.669222 9833 heartbeater.cc:511] Master 127.0.98.62:45705 requested a full tablet report, sending...
I20260619 14:47:25.671301 9645 ts_manager.cc:194] Registered new tserver with Master: 67ebec6b95b24d72bc9485ff946c58a9 (127.0.98.1:40095)
I20260619 14:47:25.672629 9645 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:33347
W20260619 14:47:25.787808 9836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:47:25.788201 9836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:47:25.788288 9836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:47:25.793587 9836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:47:25.793697 9836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:47:25.793794 9836 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 14:47:25.800279 9836 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880383132139-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880383132139-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880383132139-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45705
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9836
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880383132139-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:47:25.801569 9836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:47:25.803596 9836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:47:25.815402 9842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:47:25.815591 9836 server_base.cc:1034] running on GCE node
W20260619 14:47:25.815416 9844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:47:25.815392 9841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:47:25.816275 9836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:47:25.817041 9836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:47:25.818235 9836 hybrid_clock.cc:648] HybridClock initialized: now 1781880445818205 us; error 52 us; skew 500 ppm
I20260619 14:47:25.823959 9836 webserver.cc:460] Webserver started at http://127.0.98.2:38207/ using document root <none> and password file <none>
I20260619 14:47:25.824730 9836 fs_manager.cc:359] Metadata directory not provided
I20260619 14:47:25.824817 9836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:47:25.825089 9836 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:47:25.827334 9836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880383132139-392-0/minicluster-data/ts-1/data/instance:
uuid: "77335d5dad4f441cbd54bbc0dcc41230"
format_stamp: "Formatted at 2026-06-19 14:47:25 on dist-test-slave-2sb1"
server_key: "a3da91055a720b911724b9e91f21ee56"
server_key_iv: "3f51783ecd6e728a4f56c8c6064699b2"
server_key_version: "encryptionkey@0"
I20260619 14:47:25.827962 9836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880383132139-392-0/minicluster-data/ts-1/wal/instance:
uuid: "77335d5dad4f441cbd54bbc0dcc41230"
format_stamp: "Formatted at 2026-06-19 14:47:25 on dist-test-slave-2sb1"
server_key: "a3da91055a720b911724b9e91f21ee56"
server_key_iv: "3f51783ecd6e728a4f56c8c6064699b2"
server_key_version: "encryptionkey@0"
I20260619 14:47:25.832401 9836 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260619 14:47:25.835706 9850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:47:25.837684 9836 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:47:25.837867 9836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880383132139-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880383132139-392-0/minicluster-data/ts-1/wal
uuid: "77335d5dad4f441cbd54bbc0dcc41230"
format_stamp: "Formatted at 2026-06-19 14:47:25 on dist-test-slave-2sb1"
server_key: "a3da91055a720b911724b9e91f21ee56"
server_key_iv: "3f51783ecd6e728a4f56c8c6064699b2"
server_key_version: "encryptionkey@0"
I20260619 14:47:25.838020 9836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880383132139-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880383132139-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880383132139-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:47:25.862123 9836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:47:25.862880 9836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:47:25.863092 9836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:47:25.863729 9836 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:47:25.864804 9836 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:47:25.864856 9836 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:47:25.864919 9836 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:47:25.864954 9836 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:47:25.876307 9836 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42001
I20260619 14:47:25.876338 9963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42001 every 8 connection(s)
I20260619 14:47:25.877544 9836 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880383132139-392-0/minicluster-data/ts-1/data/info.pb
I20260619 14:47:25.881739 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 9836
I20260619 14:47:25.881875 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880383132139-392-0/minicluster-data/ts-1/wal/instance
I20260619 14:47:25.882289 392 external_mini_cluster.cc:1428] Setting key 89f0bb2f705821bb3d0e93c3350bc47c
I20260619 14:47:25.884680 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880383132139-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880383132139-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880383132139-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880383132139-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45705
--builtin_ntp_servers=127.0.98.20:39137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260619 14:47:25.890853 9684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:25.879955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54271 (local address 127.0.98.62:45705)
0619 14:47:25.880125 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:25.880129 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:25.881495 (+ 1366us) server_negotiation.cc:407] Connection header received
0619 14:47:25.883168 (+ 1673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:25.883172 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:25.883223 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:25.883292 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:25.885089 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:25.885966 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:25.888159 (+ 2193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:25.888469 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:25.888590 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:25.890244 (+ 1654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:47:25.890248 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:47:25.890250 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:47:25.890266 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:47:25.890288 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:47:25.890297 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:47:25.890374 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:25.890667 (+ 293us) server_negotiation.cc:299] Negotiation successful
0619 14:47:25.890733 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 14:47:25.892261 9966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:25.880362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45705 (local address 127.0.98.2:54271)
0619 14:47:25.881277 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:25.881323 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:25.882902 (+ 1579us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:25.883506 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:25.883518 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:25.883646 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:47:25.884807 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:25.884827 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:25.886182 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:25.886189 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:25.887947 (+ 1758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:25.887965 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:25.888753 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:25.888762 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:25.889036 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:25.889814 (+ 778us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:47:25.889817 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:47:25.889888 (+ 71us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:47:25.890007 (+ 119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:47:25.890417 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:47:25.890431 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:47:25.890441 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 14:47:25.890600 (+ 159us) client_negotiation.cc:240] Negotiation successful
0619 14:47:25.890995 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":137,"threads_started":1}
I20260619 14:47:25.893882 9964 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45705
I20260619 14:47:25.894183 9964 heartbeater.cc:464] Registering TS with master...
I20260619 14:47:25.894814 9964 heartbeater.cc:511] Master 127.0.98.62:45705 requested a full tablet report, sending...
I20260619 14:47:25.896148 9645 ts_manager.cc:194] Registered new tserver with Master: 77335d5dad4f441cbd54bbc0dcc41230 (127.0.98.2:42001)
I20260619 14:47:25.896905 9645 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:54271
W20260619 14:47:26.011925 9967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:47:26.012257 9967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:47:26.012393 9967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:47:26.016716 9967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:47:26.016857 9967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:47:26.016953 9967 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 14:47:26.023375 9967 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880383132139-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880383132139-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880383132139-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45705
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9967
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880383132139-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:47:26.024714 9967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:47:26.026230 9967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:47:26.037981 9975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:47:26.038132 9972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:47:26.037984 9973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:47:26.038623 9967 server_base.cc:1034] running on GCE node
I20260619 14:47:26.039129 9967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:47:26.039942 9967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:47:26.041154 9967 hybrid_clock.cc:648] HybridClock initialized: now 1781880446041117 us; error 61 us; skew 500 ppm
I20260619 14:47:26.043761 9967 webserver.cc:460] Webserver started at http://127.0.98.3:39053/ using document root <none> and password file <none>
I20260619 14:47:26.044477 9967 fs_manager.cc:359] Metadata directory not provided
I20260619 14:47:26.044591 9967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:47:26.044890 9967 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:47:26.047396 9967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880383132139-392-0/minicluster-data/ts-2/data/instance:
uuid: "32734d8b8dbd4a8b8de1f2bbd3abb94d"
format_stamp: "Formatted at 2026-06-19 14:47:26 on dist-test-slave-2sb1"
server_key: "88e91ede15cf03ff5c7261f31b2f40cf"
server_key_iv: "287d78d79c7ada23cce30da08804fc21"
server_key_version: "encryptionkey@0"
I20260619 14:47:26.048103 9967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880383132139-392-0/minicluster-data/ts-2/wal/instance:
uuid: "32734d8b8dbd4a8b8de1f2bbd3abb94d"
format_stamp: "Formatted at 2026-06-19 14:47:26 on dist-test-slave-2sb1"
server_key: "88e91ede15cf03ff5c7261f31b2f40cf"
server_key_iv: "287d78d79c7ada23cce30da08804fc21"
server_key_version: "encryptionkey@0"
I20260619 14:47:26.053066 9967 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260619 14:47:26.056293 9981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:47:26.058070 9967 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:47:26.058259 9967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880383132139-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880383132139-392-0/minicluster-data/ts-2/wal
uuid: "32734d8b8dbd4a8b8de1f2bbd3abb94d"
format_stamp: "Formatted at 2026-06-19 14:47:26 on dist-test-slave-2sb1"
server_key: "88e91ede15cf03ff5c7261f31b2f40cf"
server_key_iv: "287d78d79c7ada23cce30da08804fc21"
server_key_version: "encryptionkey@0"
I20260619 14:47:26.058424 9967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880383132139-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880383132139-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880383132139-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:47:26.077452 9967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:47:26.078253 9967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:47:26.078486 9967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:47:26.079124 9967 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:47:26.080123 9967 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:47:26.080200 9967 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:47:26.080266 9967 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:47:26.080323 9967 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:47:26.091637 9967 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41471
I20260619 14:47:26.091694 10094 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41471 every 8 connection(s)
I20260619 14:47:26.092730 9967 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880383132139-392-0/minicluster-data/ts-2/data/info.pb
I20260619 14:47:26.093434 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 9967
I20260619 14:47:26.093544 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781880383132139-392-0/minicluster-data/ts-2/wal/instance
I20260619 14:47:26.093858 392 external_mini_cluster.cc:1428] Setting key a2c334f43fe529d576584bd931056ae5
I20260619 14:47:26.104617 9684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:26.094890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50435 (local address 127.0.98.62:45705)
0619 14:47:26.095048 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:26.095053 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:26.096540 (+ 1487us) server_negotiation.cc:407] Connection header received
0619 14:47:26.098319 (+ 1779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:26.098325 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:26.098391 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:26.098476 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:26.100146 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:26.100791 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:26.102234 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:26.102530 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:26.102631 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:26.104059 (+ 1428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:47:26.104062 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:47:26.104065 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:47:26.104079 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:47:26.104099 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:47:26.104108 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:47:26.104179 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:26.104426 (+ 247us) server_negotiation.cc:299] Negotiation successful
0619 14:47:26.104485 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260619 14:47:26.105786 10097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:26.095449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45705 (local address 127.0.98.3:50435)
0619 14:47:26.096313 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:26.096377 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:26.098079 (+ 1702us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:26.098931 (+ 852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:26.098944 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:26.099372 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:47:26.099967 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:26.099984 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:26.100976 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:26.100980 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:26.102031 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:26.102042 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:26.102740 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:26.102743 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:26.103039 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:26.103716 (+ 677us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:47:26.103718 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:47:26.103766 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:47:26.103872 (+ 106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:47:26.104236 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:47:26.104244 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:47:26.104250 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 14:47:26.104397 (+ 147us) client_negotiation.cc:240] Negotiation successful
0619 14:47:26.104739 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":141,"threads_started":1}
I20260619 14:47:26.107507 10095 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45705
I20260619 14:47:26.107807 10095 heartbeater.cc:464] Registering TS with master...
I20260619 14:47:26.108433 10095 heartbeater.cc:511] Master 127.0.98.62:45705 requested a full tablet report, sending...
I20260619 14:47:26.109637 9645 ts_manager.cc:194] Registered new tserver with Master: 32734d8b8dbd4a8b8de1f2bbd3abb94d (127.0.98.3:41471)
I20260619 14:47:26.110378 9645 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:50435
I20260619 14:47:26.121413 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 14:47:26.124697 9684 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:47:26.123633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33432 (local address 127.0.98.62:45705)
0619 14:47:26.123814 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:26.123820 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:26.123905 (+ 85us) server_negotiation.cc:407] Connection header received
0619 14:47:26.124100 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:26.124103 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:26.124162 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:26.124260 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:26.124520 (+ 260us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33432: BlockingRecv error: recv got EOF from 127.0.0.1:33432 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260619 14:47:26.124876 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 9705
I20260619 14:47:26.130946 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 9836
I20260619 14:47:26.136375 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 9967
I20260619 14:47:26.142148 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 9614
2026-06-19T14:47:26Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (1065 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-19T14:47:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:47:26Z Disabled control of system clock
I20260619 14:47:26.172580 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880383132139-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880383132139-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880383132139-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42013
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:36555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42013
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 14:47:26.290135 10106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:47:26.290555 10106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:47:26.290599 10106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:47:26.294925 10106 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:47:26.295015 10106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:47:26.295030 10106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:47:26.295048 10106 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:47:26.295065 10106 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:47:26.300408 10106 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880383132139-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880383132139-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42013
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42013
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10106
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880383132139-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:47:26.301644 10106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:47:26.303105 10106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:47:26.312815 10114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:47:26.312866 10111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:47:26.312938 10106 server_base.cc:1034] running on GCE node
W20260619 14:47:26.312866 10112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:47:26.313735 10106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:47:26.315243 10106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:47:26.316488 10106 hybrid_clock.cc:648] HybridClock initialized: now 1781880446316438 us; error 48 us; skew 500 ppm
I20260619 14:47:26.319589 10106 webserver.cc:460] Webserver started at http://127.0.98.62:32885/ using document root <none> and password file <none>
I20260619 14:47:26.320315 10106 fs_manager.cc:359] Metadata directory not provided
I20260619 14:47:26.320400 10106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:47:26.320729 10106 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:47:26.323346 10106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880383132139-392-0/minicluster-data/master-0/data/instance:
uuid: "e6e2f3c9eed447e18e05f59f87a80ac6"
format_stamp: "Formatted at 2026-06-19 14:47:26 on dist-test-slave-2sb1"
server_key: "e4c98da47688cdacbdaac254d536f3f5"
server_key_iv: "9f37d7093e92dc69f370f1784f764789"
server_key_version: "encryptionkey@0"
I20260619 14:47:26.324031 10106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880383132139-392-0/minicluster-data/master-0/wal/instance:
uuid: "e6e2f3c9eed447e18e05f59f87a80ac6"
format_stamp: "Formatted at 2026-06-19 14:47:26 on dist-test-slave-2sb1"
server_key: "e4c98da47688cdacbdaac254d536f3f5"
server_key_iv: "9f37d7093e92dc69f370f1784f764789"
server_key_version: "encryptionkey@0"
I20260619 14:47:26.328719 10106 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.002s
I20260619 14:47:26.331765 10120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:47:26.333384 10106 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:47:26.333561 10106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880383132139-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880383132139-392-0/minicluster-data/master-0/wal
uuid: "e6e2f3c9eed447e18e05f59f87a80ac6"
format_stamp: "Formatted at 2026-06-19 14:47:26 on dist-test-slave-2sb1"
server_key: "e4c98da47688cdacbdaac254d536f3f5"
server_key_iv: "9f37d7093e92dc69f370f1784f764789"
server_key_version: "encryptionkey@0"
I20260619 14:47:26.333698 10106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880383132139-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880383132139-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880383132139-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:47:26.339027 10106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:47:26.339709 10106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:47:26.339920 10106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:47:26.347662 10106 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42013
I20260619 14:47:26.347704 10172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42013 every 8 connection(s)
I20260619 14:47:26.348824 10106 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:47:26.349402 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 10106
I20260619 14:47:26.349522 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880383132139-392-0/minicluster-data/master-0/wal/instance
I20260619 14:47:26.349813 392 external_mini_cluster.cc:1428] Setting key cee3a78e5ca2e7869780e87eff1cd9df
I20260619 14:47:26.352581 10173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:47:26.356621 10176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:26.351292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38584 (local address 127.0.98.62:42013)
0619 14:47:26.352725 (+ 1433us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:26.352753 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:26.352802 (+ 49us) server_negotiation.cc:407] Connection header received
0619 14:47:26.353482 (+ 680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:26.353509 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:26.353880 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:26.354272 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:26.355239 (+ 967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:47:26.355244 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:47:26.355258 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:47:26.355270 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:47:26.355328 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:47:26.355345 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:47:26.355521 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:26.355594 (+ 73us) server_negotiation.cc:299] Negotiation successful
0619 14:47:26.355835 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":923,"thread_start_us":714,"threads_started":1}
I20260619 14:47:26.359483 10173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6e2f3c9eed447e18e05f59f87a80ac6: Bootstrap starting.
I20260619 14:47:26.361953 10173 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e6e2f3c9eed447e18e05f59f87a80ac6: Neither blocks nor log segments found. Creating new log.
I20260619 14:47:26.362785 10173 log.cc:824] T 00000000000000000000000000000000 P e6e2f3c9eed447e18e05f59f87a80ac6: Log is configured to *not* fsync() on all Append() calls
I20260619 14:47:26.365540 10173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6e2f3c9eed447e18e05f59f87a80ac6: No bootstrap required, opened a new log
I20260619 14:47:26.369763 10173 raft_consensus.cc:348] T 00000000000000000000000000000000 P e6e2f3c9eed447e18e05f59f87a80ac6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6e2f3c9eed447e18e05f59f87a80ac6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42013 } }
I20260619 14:47:26.370122 10173 raft_consensus.cc:374] T 00000000000000000000000000000000 P e6e2f3c9eed447e18e05f59f87a80ac6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:47:26.370184 10173 raft_consensus.cc:729] T 00000000000000000000000000000000 P e6e2f3c9eed447e18e05f59f87a80ac6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6e2f3c9eed447e18e05f59f87a80ac6, State: Initialized, Role: FOLLOWER
I20260619 14:47:26.370728 10173 consensus_queue.cc:260] T 00000000000000000000000000000000 P e6e2f3c9eed447e18e05f59f87a80ac6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6e2f3c9eed447e18e05f59f87a80ac6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42013 } }
I20260619 14:47:26.370945 10173 raft_consensus.cc:388] T 00000000000000000000000000000000 P e6e2f3c9eed447e18e05f59f87a80ac6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:47:26.371022 10173 raft_consensus.cc:482] T 00000000000000000000000000000000 P e6e2f3c9eed447e18e05f59f87a80ac6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:47:26.371174 10173 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e6e2f3c9eed447e18e05f59f87a80ac6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:47:26.372308 10173 raft_consensus.cc:504] T 00000000000000000000000000000000 P e6e2f3c9eed447e18e05f59f87a80ac6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6e2f3c9eed447e18e05f59f87a80ac6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42013 } }
I20260619 14:47:26.372812 10173 leader_election.cc:302] T 00000000000000000000000000000000 P e6e2f3c9eed447e18e05f59f87a80ac6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e6e2f3c9eed447e18e05f59f87a80ac6; no voters:
I20260619 14:47:26.373243 10173 leader_election.cc:288] T 00000000000000000000000000000000 P e6e2f3c9eed447e18e05f59f87a80ac6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:47:26.373611 10178 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e6e2f3c9eed447e18e05f59f87a80ac6 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:47:26.373922 10178 raft_consensus.cc:686] T 00000000000000000000000000000000 P e6e2f3c9eed447e18e05f59f87a80ac6 [term 1 LEADER]: Becoming Leader. State: Replica: e6e2f3c9eed447e18e05f59f87a80ac6, State: Running, Role: LEADER
I20260619 14:47:26.374331 10178 consensus_queue.cc:237] T 00000000000000000000000000000000 P e6e2f3c9eed447e18e05f59f87a80ac6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6e2f3c9eed447e18e05f59f87a80ac6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42013 } }
I20260619 14:47:26.374756 10173 sys_catalog.cc:564] T 00000000000000000000000000000000 P e6e2f3c9eed447e18e05f59f87a80ac6 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:47:26.376286 10180 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6e2f3c9eed447e18e05f59f87a80ac6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e6e2f3c9eed447e18e05f59f87a80ac6. Latest consensus state: current_term: 1 leader_uuid: "e6e2f3c9eed447e18e05f59f87a80ac6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6e2f3c9eed447e18e05f59f87a80ac6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42013 } } }
I20260619 14:47:26.376286 10179 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6e2f3c9eed447e18e05f59f87a80ac6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e6e2f3c9eed447e18e05f59f87a80ac6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6e2f3c9eed447e18e05f59f87a80ac6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42013 } } }
I20260619 14:47:26.376457 10180 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6e2f3c9eed447e18e05f59f87a80ac6 [sys.catalog]: This master's current role is: LEADER
I20260619 14:47:26.376479 10179 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6e2f3c9eed447e18e05f59f87a80ac6 [sys.catalog]: This master's current role is: LEADER
I20260619 14:47:26.377106 10188 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:47:26.380414 10188 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:47:26.387750 10188 catalog_manager.cc:1347] Generated new cluster ID: 09f49ea9bc4a4f9ca568d0e34b79f0c3
I20260619 14:47:26.387869 10188 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:47:26.417400 10188 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:47:26.417838 10188 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:47:26.423158 10188 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e6e2f3c9eed447e18e05f59f87a80ac6: Generated new TSK 0
I20260619 14:47:26.424064 10188 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:47:26.432483 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880383132139-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880383132139-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880383132139-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880383132139-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42013
--builtin_ntp_servers=127.0.98.20:36555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260619 14:47:26.556133 10197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:47:26.556443 10197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:47:26.556486 10197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:47:26.560802 10197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:47:26.560879 10197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:47:26.560940 10197 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 14:47:26.565858 10197 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880383132139-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880383132139-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880383132139-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42013
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10197
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880383132139-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:47:26.566960 10197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:47:26.568395 10197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:47:26.580077 10203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:47:26.580098 10205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:47:26.580173 10202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:47:26.580269 10197 server_base.cc:1034] running on GCE node
I20260619 14:47:26.580885 10197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:47:26.581593 10197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:47:26.582787 10197 hybrid_clock.cc:648] HybridClock initialized: now 1781880446582758 us; error 45 us; skew 500 ppm
I20260619 14:47:26.585259 10197 webserver.cc:460] Webserver started at http://127.0.98.1:45843/ using document root <none> and password file <none>
I20260619 14:47:26.585961 10197 fs_manager.cc:359] Metadata directory not provided
I20260619 14:47:26.586035 10197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:47:26.586262 10197 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:47:26.588843 10197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880383132139-392-0/minicluster-data/ts-0/data/instance:
uuid: "56b68f75f8e44e45a2dd3f5ca84d28e0"
format_stamp: "Formatted at 2026-06-19 14:47:26 on dist-test-slave-2sb1"
server_key: "c9e56a73fb966864eca8ef714df65331"
server_key_iv: "29baaedd7cc90d0c81d8a40b337d028a"
server_key_version: "encryptionkey@0"
I20260619 14:47:26.589634 10197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880383132139-392-0/minicluster-data/ts-0/wal/instance:
uuid: "56b68f75f8e44e45a2dd3f5ca84d28e0"
format_stamp: "Formatted at 2026-06-19 14:47:26 on dist-test-slave-2sb1"
server_key: "c9e56a73fb966864eca8ef714df65331"
server_key_iv: "29baaedd7cc90d0c81d8a40b337d028a"
server_key_version: "encryptionkey@0"
I20260619 14:47:26.594302 10197 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 14:47:26.597230 10211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:47:26.599224 10197 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:47:26.599377 10197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880383132139-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880383132139-392-0/minicluster-data/ts-0/wal
uuid: "56b68f75f8e44e45a2dd3f5ca84d28e0"
format_stamp: "Formatted at 2026-06-19 14:47:26 on dist-test-slave-2sb1"
server_key: "c9e56a73fb966864eca8ef714df65331"
server_key_iv: "29baaedd7cc90d0c81d8a40b337d028a"
server_key_version: "encryptionkey@0"
I20260619 14:47:26.599496 10197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880383132139-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880383132139-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880383132139-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:47:26.605563 10197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:47:26.606274 10197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:47:26.606484 10197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:47:26.607092 10197 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:47:26.608067 10197 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:47:26.608116 10197 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:47:26.608156 10197 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:47:26.608172 10197 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:47:26.618122 10197 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33451
I20260619 14:47:26.618144 10324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33451 every 8 connection(s)
I20260619 14:47:26.619108 10197 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880383132139-392-0/minicluster-data/ts-0/data/info.pb
I20260619 14:47:26.620167 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 10197
I20260619 14:47:26.620254 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880383132139-392-0/minicluster-data/ts-0/wal/instance
I20260619 14:47:26.620595 392 external_mini_cluster.cc:1428] Setting key e3cf4059d1bc424ec682c55b67dc791b
I20260619 14:47:26.622906 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880383132139-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880383132139-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880383132139-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880383132139-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42013
--builtin_ntp_servers=127.0.98.20:36555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260619 14:47:26.626081 10176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:26.621229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33645 (local address 127.0.98.62:42013)
0619 14:47:26.621506 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:26.621510 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:26.622774 (+ 1264us) server_negotiation.cc:407] Connection header received
0619 14:47:26.624136 (+ 1362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:26.624139 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:26.624183 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:26.624262 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:26.625576 (+ 1314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:47:26.625579 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:47:26.625582 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:47:26.625596 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:47:26.625619 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:47:26.625629 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:47:26.625736 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:26.625883 (+ 147us) server_negotiation.cc:299] Negotiation successful
0619 14:47:26.625940 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190}
I20260619 14:47:26.627149 10327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:26.621752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42013 (local address 127.0.98.1:33645)
0619 14:47:26.622590 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:26.622645 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:26.623905 (+ 1260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:26.624439 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:26.624445 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:26.624542 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:47:26.625278 (+ 736us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:47:26.625280 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:47:26.625324 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:47:26.625407 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:47:26.625727 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:47:26.625735 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:47:26.625741 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 14:47:26.625934 (+ 193us) client_negotiation.cc:240] Negotiation successful
0619 14:47:26.626278 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":194,"threads_started":1}
I20260619 14:47:26.628785 10325 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42013
I20260619 14:47:26.629053 10325 heartbeater.cc:464] Registering TS with master...
I20260619 14:47:26.629514 10325 heartbeater.cc:511] Master 127.0.98.62:42013 requested a full tablet report, sending...
I20260619 14:47:26.631505 10137 ts_manager.cc:194] Registered new tserver with Master: 56b68f75f8e44e45a2dd3f5ca84d28e0 (127.0.98.1:33451)
W20260619 14:47:26.746330 10328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:47:26.746630 10328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:47:26.746677 10328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:47:26.751189 10328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:47:26.751271 10328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:47:26.751333 10328 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 14:47:26.756434 10328 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880383132139-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880383132139-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880383132139-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42013
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10328
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880383132139-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:47:26.757593 10328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:47:26.758996 10328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:47:26.769697 10336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:47:26.769697 10333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:47:26.769831 10334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:47:26.769893 10328 server_base.cc:1034] running on GCE node
I20260619 14:47:26.770568 10328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:47:26.771267 10328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:47:26.772490 10328 hybrid_clock.cc:648] HybridClock initialized: now 1781880446772467 us; error 41 us; skew 500 ppm
I20260619 14:47:26.774847 10328 webserver.cc:460] Webserver started at http://127.0.98.2:46427/ using document root <none> and password file <none>
I20260619 14:47:26.775772 10328 fs_manager.cc:359] Metadata directory not provided
I20260619 14:47:26.775873 10328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:47:26.776104 10328 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:47:26.778522 10328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880383132139-392-0/minicluster-data/ts-1/data/instance:
uuid: "c0ce36564f2a401a917769bd42245ae1"
format_stamp: "Formatted at 2026-06-19 14:47:26 on dist-test-slave-2sb1"
server_key: "7379c3e432e11c25e393855ad35b1297"
server_key_iv: "63435b4f2d5a6040f6a44e7c86212eec"
server_key_version: "encryptionkey@0"
I20260619 14:47:26.779163 10328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880383132139-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c0ce36564f2a401a917769bd42245ae1"
format_stamp: "Formatted at 2026-06-19 14:47:26 on dist-test-slave-2sb1"
server_key: "7379c3e432e11c25e393855ad35b1297"
server_key_iv: "63435b4f2d5a6040f6a44e7c86212eec"
server_key_version: "encryptionkey@0"
I20260619 14:47:26.783556 10328 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 14:47:26.786377 10342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:47:26.787757 10328 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 14:47:26.787919 10328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880383132139-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880383132139-392-0/minicluster-data/ts-1/wal
uuid: "c0ce36564f2a401a917769bd42245ae1"
format_stamp: "Formatted at 2026-06-19 14:47:26 on dist-test-slave-2sb1"
server_key: "7379c3e432e11c25e393855ad35b1297"
server_key_iv: "63435b4f2d5a6040f6a44e7c86212eec"
server_key_version: "encryptionkey@0"
I20260619 14:47:26.788054 10328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880383132139-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880383132139-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880383132139-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:47:26.793089 10328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:47:26.793798 10328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:47:26.793998 10328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:47:26.794574 10328 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:47:26.795535 10328 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:47:26.795610 10328 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:47:26.795673 10328 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:47:26.795717 10328 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:47:26.805132 10328 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38331
I20260619 14:47:26.805343 10455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38331 every 8 connection(s)
I20260619 14:47:26.806161 10328 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880383132139-392-0/minicluster-data/ts-1/data/info.pb
I20260619 14:47:26.810287 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 10328
I20260619 14:47:26.810492 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880383132139-392-0/minicluster-data/ts-1/wal/instance
I20260619 14:47:26.810803 392 external_mini_cluster.cc:1428] Setting key 5953e9ce18cb360fc9b9af70f97138bd
I20260619 14:47:26.813362 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880383132139-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880383132139-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880383132139-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880383132139-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42013
--builtin_ntp_servers=127.0.98.20:36555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260619 14:47:26.814118 10176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:26.808342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60895 (local address 127.0.98.62:42013)
0619 14:47:26.808529 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:26.808534 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:26.809717 (+ 1183us) server_negotiation.cc:407] Connection header received
0619 14:47:26.811317 (+ 1600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:26.811320 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:26.811390 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:26.811492 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:26.813278 (+ 1786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:47:26.813282 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:47:26.813285 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:47:26.813302 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:47:26.813329 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:47:26.813341 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:47:26.813556 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:26.813867 (+ 311us) server_negotiation.cc:299] Negotiation successful
0619 14:47:26.813924 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 14:47:26.815131 10458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:26.808830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42013 (local address 127.0.98.2:60895)
0619 14:47:26.809549 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:26.809592 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:26.811050 (+ 1458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:26.811800 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:26.811810 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:26.811932 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:47:26.812952 (+ 1020us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:47:26.812954 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:47:26.813019 (+ 65us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:47:26.813100 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:47:26.813666 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:47:26.813679 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:47:26.813688 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 14:47:26.813878 (+ 190us) client_negotiation.cc:240] Negotiation successful
0619 14:47:26.814223 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"thread_start_us":132,"threads_started":1}
I20260619 14:47:26.816619 10456 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42013
I20260619 14:47:26.816919 10456 heartbeater.cc:464] Registering TS with master...
I20260619 14:47:26.817373 10456 heartbeater.cc:511] Master 127.0.98.62:42013 requested a full tablet report, sending...
I20260619 14:47:26.818511 10137 ts_manager.cc:194] Registered new tserver with Master: c0ce36564f2a401a917769bd42245ae1 (127.0.98.2:38331)
W20260619 14:47:26.931391 10459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:47:26.931681 10459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:47:26.931725 10459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:47:26.935948 10459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:47:26.936028 10459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:47:26.936086 10459 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 14:47:26.941959 10459 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880383132139-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880383132139-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880383132139-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42013
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10459
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880383132139-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:47:26.943162 10459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:47:26.944614 10459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:47:26.955026 10465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:47:26.955039 10464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:47:26.955057 10467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:47:26.955277 10459 server_base.cc:1034] running on GCE node
I20260619 14:47:26.955854 10459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:47:26.956635 10459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:47:26.958000 10459 hybrid_clock.cc:648] HybridClock initialized: now 1781880446957970 us; error 45 us; skew 500 ppm
I20260619 14:47:26.960528 10459 webserver.cc:460] Webserver started at http://127.0.98.3:36553/ using document root <none> and password file <none>
I20260619 14:47:26.961234 10459 fs_manager.cc:359] Metadata directory not provided
I20260619 14:47:26.961344 10459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:47:26.961592 10459 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:47:26.963830 10459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880383132139-392-0/minicluster-data/ts-2/data/instance:
uuid: "203b7cad64b74580aa78ffa51f55f873"
format_stamp: "Formatted at 2026-06-19 14:47:26 on dist-test-slave-2sb1"
server_key: "88e3d99177281ad2e18ea38a493cd34a"
server_key_iv: "d678ca54ff190382cd2447f8c69c2d9d"
server_key_version: "encryptionkey@0"
I20260619 14:47:26.964483 10459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880383132139-392-0/minicluster-data/ts-2/wal/instance:
uuid: "203b7cad64b74580aa78ffa51f55f873"
format_stamp: "Formatted at 2026-06-19 14:47:26 on dist-test-slave-2sb1"
server_key: "88e3d99177281ad2e18ea38a493cd34a"
server_key_iv: "d678ca54ff190382cd2447f8c69c2d9d"
server_key_version: "encryptionkey@0"
I20260619 14:47:26.969000 10459 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 14:47:26.972128 10473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:47:26.973786 10459 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:47:26.973960 10459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880383132139-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880383132139-392-0/minicluster-data/ts-2/wal
uuid: "203b7cad64b74580aa78ffa51f55f873"
format_stamp: "Formatted at 2026-06-19 14:47:26 on dist-test-slave-2sb1"
server_key: "88e3d99177281ad2e18ea38a493cd34a"
server_key_iv: "d678ca54ff190382cd2447f8c69c2d9d"
server_key_version: "encryptionkey@0"
I20260619 14:47:26.974116 10459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880383132139-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880383132139-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880383132139-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:47:26.979490 10459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:47:26.980177 10459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:47:26.980377 10459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:47:26.981109 10459 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:47:26.982091 10459 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:47:26.982169 10459 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:47:26.982237 10459 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:47:26.982285 10459 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:47:26.992417 10459 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34023
I20260619 14:47:26.992628 10586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34023 every 8 connection(s)
I20260619 14:47:26.993563 10459 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880383132139-392-0/minicluster-data/ts-2/data/info.pb
I20260619 14:47:27.000730 10176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:26.995598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58977 (local address 127.0.98.62:42013)
0619 14:47:26.995772 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:26.995776 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:26.997132 (+ 1356us) server_negotiation.cc:407] Connection header received
0619 14:47:26.998644 (+ 1512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:26.998648 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:26.998706 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:26.998802 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:27.000145 (+ 1343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:47:27.000149 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:47:27.000152 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:47:27.000171 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:47:27.000191 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:47:27.000199 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:47:27.000307 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:27.000499 (+ 192us) server_negotiation.cc:299] Negotiation successful
0619 14:47:27.000546 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 14:47:27.001103 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 10459
I20260619 14:47:27.001230 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781880383132139-392-0/minicluster-data/ts-2/wal/instance
I20260619 14:47:27.001555 392 external_mini_cluster.cc:1428] Setting key a2c9f3bb5d0230f8cba489a06316f960
I20260619 14:47:27.002087 10589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:26.996064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42013 (local address 127.0.98.3:58977)
0619 14:47:26.996920 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:26.996987 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:26.998361 (+ 1374us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:26.999004 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:26.999012 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:26.999108 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:47:26.999808 (+ 700us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:47:26.999810 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:47:26.999870 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:47:26.999956 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:47:27.000338 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:47:27.000353 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:47:27.000359 (+ 6us) client_negotiation.cc:769] Sending connection context
0619 14:47:27.000542 (+ 183us) client_negotiation.cc:240] Negotiation successful
0619 14:47:27.000824 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":132,"threads_started":1}
I20260619 14:47:27.003412 10587 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42013
I20260619 14:47:27.003823 10587 heartbeater.cc:464] Registering TS with master...
I20260619 14:47:27.004284 10587 heartbeater.cc:511] Master 127.0.98.62:42013 requested a full tablet report, sending...
I20260619 14:47:27.005471 10137 ts_manager.cc:194] Registered new tserver with Master: 203b7cad64b74580aa78ffa51f55f873 (127.0.98.3:34023)
I20260619 14:47:27.016403 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260619 14:47:27.019707 10176 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:47:27.018637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38588 (local address 127.0.98.62:42013)
0619 14:47:27.018895 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:27.018900 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:27.019047 (+ 147us) server_negotiation.cc:407] Connection header received
0619 14:47:27.019206 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:27.019209 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:27.019250 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:27.019349 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:27.019597 (+ 248us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38588: BlockingRecv error: recv got EOF from 127.0.0.1:38588 (error 108)
Metrics: {"server-negotiator.queue_time_us":150}
I20260619 14:47:27.020257 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 10197
I20260619 14:47:27.025619 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 10328
I20260619 14:47:27.031176 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 10459
I20260619 14:47:27.036811 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 10106
2026-06-19T14:47:27Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (896 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880383132139-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:47:27 dist-test-slave-2sb1 krb5kdc[10597](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:47:27 dist-test-slave-2sb1 krb5kdc[10597](info): set up 2 sockets
Jun 19 14:47:27 dist-test-slave-2sb1 krb5kdc[10597](info): commencing operation
krb5kdc: starting...
W20260619 14:47:28.154649 392 mini_kdc.cc:121] Time spent starting KDC: real 1.094s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880383132139-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880383132139-392-0/krb5kdc/test-user.keytab.
Jun 19 14:47:28 dist-test-slave-2sb1 krb5kdc[10597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880448, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:47:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:47:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 14:47:28.297580 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880383132139-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880383132139-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880383132139-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42553
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42553
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:47:28.419870 10610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:47:28.420408 10610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:47:28.420521 10610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:47:28.425374 10610 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:47:28.425447 10610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:47:28.425468 10610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:47:28.425485 10610 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:47:28.425501 10610 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:47:28.431310 10610 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880383132139-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880383132139-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42553
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42553
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10610
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880383132139-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:47:28.432530 10610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:47:28.433655 10610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:47:28.443742 10615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:47:28.443775 10618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:47:28.443742 10616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:47:28.443809 10610 server_base.cc:1034] running on GCE node
I20260619 14:47:28.444785 10610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:47:28.445992 10610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:47:28.447233 10610 hybrid_clock.cc:648] HybridClock initialized: now 1781880448447208 us; error 40 us; skew 500 ppm
Jun 19 14:47:28 dist-test-slave-2sb1 krb5kdc[10597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880448, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:47:28.450641 10610 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 14:47:28.451656 10610 webserver.cc:460] Webserver started at http://127.0.98.62:35861/ using document root <none> and password file <none>
I20260619 14:47:28.452235 10610 fs_manager.cc:359] Metadata directory not provided
I20260619 14:47:28.452299 10610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:47:28.452488 10610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:47:28.454730 10610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880383132139-392-0/minicluster-data/master-0/data/instance:
uuid: "ecde9ecb467c44d8ac03c9ad2971b088"
format_stamp: "Formatted at 2026-06-19 14:47:28 on dist-test-slave-2sb1"
server_key: "550b96416477d43be0245abd4746b0a3"
server_key_iv: "42fd36171c9138cdfe54f1063319336a"
server_key_version: "encryptionkey@0"
I20260619 14:47:28.455334 10610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880383132139-392-0/minicluster-data/master-0/wal/instance:
uuid: "ecde9ecb467c44d8ac03c9ad2971b088"
format_stamp: "Formatted at 2026-06-19 14:47:28 on dist-test-slave-2sb1"
server_key: "550b96416477d43be0245abd4746b0a3"
server_key_iv: "42fd36171c9138cdfe54f1063319336a"
server_key_version: "encryptionkey@0"
I20260619 14:47:28.459709 10610 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20260619 14:47:28.462774 10625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:47:28.464507 10610 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 14:47:28.464747 10610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880383132139-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880383132139-392-0/minicluster-data/master-0/wal
uuid: "ecde9ecb467c44d8ac03c9ad2971b088"
format_stamp: "Formatted at 2026-06-19 14:47:28 on dist-test-slave-2sb1"
server_key: "550b96416477d43be0245abd4746b0a3"
server_key_iv: "42fd36171c9138cdfe54f1063319336a"
server_key_version: "encryptionkey@0"
I20260619 14:47:28.464887 10610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880383132139-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880383132139-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880383132139-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:47:28.479275 10610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:47:28.483804 10610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:47:28.484009 10610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:47:28.491621 10610 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42553
I20260619 14:47:28.491963 10677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42553 every 8 connection(s)
I20260619 14:47:28.492743 10610 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:47:28.495564 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 10610
I20260619 14:47:28.495672 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880383132139-392-0/minicluster-data/master-0/wal/instance
I20260619 14:47:28.496037 392 external_mini_cluster.cc:1428] Setting key 7f21bc6b4e5dfe11ca0e70976d6c9a89
I20260619 14:47:28.496294 10678 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:47:28.504241 10678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecde9ecb467c44d8ac03c9ad2971b088: Bootstrap starting.
Jun 19 14:47:28 dist-test-slave-2sb1 krb5kdc[10597](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880448, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:47:28.507304 10678 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ecde9ecb467c44d8ac03c9ad2971b088: Neither blocks nor log segments found. Creating new log.
I20260619 14:47:28.508246 10678 log.cc:824] T 00000000000000000000000000000000 P ecde9ecb467c44d8ac03c9ad2971b088: Log is configured to *not* fsync() on all Append() calls
I20260619 14:47:28.510753 10678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecde9ecb467c44d8ac03c9ad2971b088: No bootstrap required, opened a new log
I20260619 14:47:28.513043 10681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:28.497563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50634 (local address 127.0.98.62:42553)
0619 14:47:28.498016 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:28.498031 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:28.498068 (+ 37us) server_negotiation.cc:407] Connection header received
0619 14:47:28.498734 (+ 666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:28.498754 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:28.499122 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:28.499525 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:28.500554 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:28.501930 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:28.503190 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:28.503736 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:28.503888 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:28.506893 (+ 3005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:47:28.506900 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:47:28.506918 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:47:28.506956 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:47:28.509455 (+ 2499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:28.510229 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:28.510235 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:28.510240 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:28.510392 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:28.510672 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:28.510675 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:28.510676 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:28.511192 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:47:28.511391 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:28.511698 (+ 307us) server_negotiation.cc:299] Negotiation successful
0619 14:47:28.511999 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":156,"threads_started":1}
I20260619 14:47:28.515244 10678 raft_consensus.cc:348] T 00000000000000000000000000000000 P ecde9ecb467c44d8ac03c9ad2971b088 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecde9ecb467c44d8ac03c9ad2971b088" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42553 } }
I20260619 14:47:28.515496 10678 raft_consensus.cc:374] T 00000000000000000000000000000000 P ecde9ecb467c44d8ac03c9ad2971b088 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:47:28.515532 10678 raft_consensus.cc:729] T 00000000000000000000000000000000 P ecde9ecb467c44d8ac03c9ad2971b088 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecde9ecb467c44d8ac03c9ad2971b088, State: Initialized, Role: FOLLOWER
I20260619 14:47:28.516031 10678 consensus_queue.cc:260] T 00000000000000000000000000000000 P ecde9ecb467c44d8ac03c9ad2971b088 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecde9ecb467c44d8ac03c9ad2971b088" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42553 } }
I20260619 14:47:28.516172 10678 raft_consensus.cc:388] T 00000000000000000000000000000000 P ecde9ecb467c44d8ac03c9ad2971b088 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:47:28.516214 10678 raft_consensus.cc:482] T 00000000000000000000000000000000 P ecde9ecb467c44d8ac03c9ad2971b088 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:47:28.516281 10678 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ecde9ecb467c44d8ac03c9ad2971b088 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:47:28.517390 10678 raft_consensus.cc:504] T 00000000000000000000000000000000 P ecde9ecb467c44d8ac03c9ad2971b088 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecde9ecb467c44d8ac03c9ad2971b088" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42553 } }
I20260619 14:47:28.517731 10678 leader_election.cc:302] T 00000000000000000000000000000000 P ecde9ecb467c44d8ac03c9ad2971b088 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ecde9ecb467c44d8ac03c9ad2971b088; no voters:
I20260619 14:47:28.518079 10678 leader_election.cc:288] T 00000000000000000000000000000000 P ecde9ecb467c44d8ac03c9ad2971b088 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:47:28.518164 10683 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ecde9ecb467c44d8ac03c9ad2971b088 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:47:28.518373 10683 raft_consensus.cc:686] T 00000000000000000000000000000000 P ecde9ecb467c44d8ac03c9ad2971b088 [term 1 LEADER]: Becoming Leader. State: Replica: ecde9ecb467c44d8ac03c9ad2971b088, State: Running, Role: LEADER
I20260619 14:47:28.518688 10683 consensus_queue.cc:237] T 00000000000000000000000000000000 P ecde9ecb467c44d8ac03c9ad2971b088 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecde9ecb467c44d8ac03c9ad2971b088" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42553 } }
I20260619 14:47:28.519512 10678 sys_catalog.cc:564] T 00000000000000000000000000000000 P ecde9ecb467c44d8ac03c9ad2971b088 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:47:28.520511 10685 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecde9ecb467c44d8ac03c9ad2971b088 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ecde9ecb467c44d8ac03c9ad2971b088" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecde9ecb467c44d8ac03c9ad2971b088" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42553 } } }
I20260619 14:47:28.520824 10685 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecde9ecb467c44d8ac03c9ad2971b088 [sys.catalog]: This master's current role is: LEADER
I20260619 14:47:28.521171 10684 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecde9ecb467c44d8ac03c9ad2971b088 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ecde9ecb467c44d8ac03c9ad2971b088. Latest consensus state: current_term: 1 leader_uuid: "ecde9ecb467c44d8ac03c9ad2971b088" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecde9ecb467c44d8ac03c9ad2971b088" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42553 } } }
I20260619 14:47:28.521538 10684 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecde9ecb467c44d8ac03c9ad2971b088 [sys.catalog]: This master's current role is: LEADER
W20260619 14:47:28.525074 10699 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ecde9ecb467c44d8ac03c9ad2971b088: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:47:28.525154 10699 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:47:28.525249 10692 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:47:28.526031 10692 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:47:28.532521 10692 catalog_manager.cc:1347] Generated new cluster ID: 42736f5b896843c3ad8e83af8aa465c8
I20260619 14:47:28.532612 10692 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:47:28.545112 10692 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:47:28.546309 10692 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:47:28.561264 10692 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ecde9ecb467c44d8ac03c9ad2971b088: Generated new TSK 0
I20260619 14:47:28.562348 10692 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880383132139-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880383132139-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 14:47:28.608909 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880383132139-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880383132139-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880383132139-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880383132139-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42553
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880383132139-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:47:28.733412 10704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:47:28.733713 10704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:47:28.733759 10704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:47:28.738302 10704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:47:28.738394 10704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:47:28.738456 10704 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 14:47:28.744123 10704 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880383132139-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880383132139-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880383132139-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880383132139-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42553
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10704
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880383132139-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:47:28.745321 10704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:47:28.746438 10704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:47:28.757485 10712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:47:28.757445 10710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:47:28.757445 10709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:47:28.757695 10704 server_base.cc:1034] running on GCE node
I20260619 14:47:28.758340 10704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:47:28.759087 10704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:47:28.760314 10704 hybrid_clock.cc:648] HybridClock initialized: now 1781880448760279 us; error 50 us; skew 500 ppm
Jun 19 14:47:28 dist-test-slave-2sb1 krb5kdc[10597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880448, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:47:28.763857 10704 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 14:47:28.765123 10704 webserver.cc:460] Webserver started at http://127.0.98.1:35057/ using document root <none> and password file <none>
I20260619 14:47:28.765739 10704 fs_manager.cc:359] Metadata directory not provided
I20260619 14:47:28.765841 10704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:47:28.766145 10704 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:47:28.768412 10704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880383132139-392-0/minicluster-data/ts-0/data/instance:
uuid: "8e94a8bcf0e84520aef549e11d72c6f3"
format_stamp: "Formatted at 2026-06-19 14:47:28 on dist-test-slave-2sb1"
server_key: "fc28cc70b6476e329f108fa242c1be6a"
server_key_iv: "2c965c344c577a962a1a9e519669f28d"
server_key_version: "encryptionkey@0"
I20260619 14:47:28.769129 10704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880383132139-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8e94a8bcf0e84520aef549e11d72c6f3"
format_stamp: "Formatted at 2026-06-19 14:47:28 on dist-test-slave-2sb1"
server_key: "fc28cc70b6476e329f108fa242c1be6a"
server_key_iv: "2c965c344c577a962a1a9e519669f28d"
server_key_version: "encryptionkey@0"
I20260619 14:47:28.773458 10704 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 14:47:28.776983 10719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:47:28.778544 10704 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:47:28.778720 10704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880383132139-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880383132139-392-0/minicluster-data/ts-0/wal
uuid: "8e94a8bcf0e84520aef549e11d72c6f3"
format_stamp: "Formatted at 2026-06-19 14:47:28 on dist-test-slave-2sb1"
server_key: "fc28cc70b6476e329f108fa242c1be6a"
server_key_iv: "2c965c344c577a962a1a9e519669f28d"
server_key_version: "encryptionkey@0"
I20260619 14:47:28.778863 10704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880383132139-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880383132139-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880383132139-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:47:28.806217 10704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:47:28.810787 10704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:47:28.811059 10704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:47:28.811769 10704 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:47:28.812888 10704 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:47:28.813081 10704 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:47:28.813170 10704 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:47:28.813220 10704 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:47:28.826195 10704 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34983
I20260619 14:47:28.826212 10832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34983 every 8 connection(s)
I20260619 14:47:28.827378 10704 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880383132139-392-0/minicluster-data/ts-0/data/info.pb
I20260619 14:47:28.837459 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 10704
I20260619 14:47:28.837803 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880383132139-392-0/minicluster-data/ts-0/wal/instance
I20260619 14:47:28.838325 392 external_mini_cluster.cc:1428] Setting key d602e65a9c6d4418b53aa58868eb9440
Jun 19 14:47:28 dist-test-slave-2sb1 krb5kdc[10597](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880448, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:47:28.845324 10681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:28.829540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45805 (local address 127.0.98.62:42553)
0619 14:47:28.829748 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:28.829752 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:28.831024 (+ 1272us) server_negotiation.cc:407] Connection header received
0619 14:47:28.832061 (+ 1037us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:28.832064 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:28.832115 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:28.832210 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:28.833843 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:28.834482 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:28.836095 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:28.836341 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:28.836544 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:28.840528 (+ 3984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:47:28.840531 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:47:28.840533 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:47:28.840561 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:47:28.842338 (+ 1777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:28.843132 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:28.843139 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:28.843144 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:28.843247 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:28.843668 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:28.843674 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:28.843679 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:28.844121 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:47:28.844235 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:28.844924 (+ 689us) server_negotiation.cc:299] Negotiation successful
0619 14:47:28.845110 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260619 14:47:28.846213 10835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:28.829945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42553 (local address 127.0.98.1:45805)
0619 14:47:28.830772 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:28.830822 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:28.831717 (+ 895us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:28.832596 (+ 879us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:28.832604 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:28.833123 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:47:28.833664 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:28.833679 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:28.834660 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:28.834663 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:28.835767 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:28.835780 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:28.836781 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:28.836786 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:28.836971 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:28.837898 (+ 927us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:47:28.837923 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:47:28.840246 (+ 2323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:47:28.842495 (+ 2249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:47:28.842506 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:47:28.842529 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:47:28.842975 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:47:28.843365 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:47:28.843367 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:47:28.843369 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:47:28.843551 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:47:28.844273 (+ 722us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:47:28.844278 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:47:28.844554 (+ 276us) client_negotiation.cc:769] Sending connection context
0619 14:47:28.844831 (+ 277us) client_negotiation.cc:240] Negotiation successful
0619 14:47:28.845260 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"thread_start_us":132,"threads_started":1}
I20260619 14:47:28.847666 10833 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42553
I20260619 14:47:28.848057 10833 heartbeater.cc:464] Registering TS with master...
I20260619 14:47:28.848762 10833 heartbeater.cc:511] Master 127.0.98.62:42553 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 14:47:28.850696 10642 ts_manager.cc:194] Registered new tserver with Master: 8e94a8bcf0e84520aef549e11d72c6f3 (127.0.98.1:34983)
I20260619 14:47:28.851964 10642 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45805
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880383132139-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880383132139-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 14:47:28.873406 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880383132139-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880383132139-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880383132139-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880383132139-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42553
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880383132139-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:47:28.990414 10838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:47:28.990761 10838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:47:28.990841 10838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:47:28.995292 10838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:47:28.995422 10838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:47:28.995527 10838 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 14:47:29.001960 10838 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880383132139-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880383132139-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880383132139-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880383132139-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42553
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10838
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880383132139-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:47:29.003162 10838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:47:29.004278 10838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:47:29.015388 10846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:47:29.015408 10843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:47:29.015388 10844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:47:29.015642 10838 server_base.cc:1034] running on GCE node
I20260619 14:47:29.016376 10838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:47:29.017261 10838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:47:29.018605 10838 hybrid_clock.cc:648] HybridClock initialized: now 1781880449018570 us; error 55 us; skew 500 ppm
Jun 19 14:47:29 dist-test-slave-2sb1 krb5kdc[10597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880449, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:47:29.023324 10838 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 14:47:29.024531 10838 webserver.cc:460] Webserver started at http://127.0.98.2:35185/ using document root <none> and password file <none>
I20260619 14:47:29.025230 10838 fs_manager.cc:359] Metadata directory not provided
I20260619 14:47:29.025341 10838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:47:29.025625 10838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:47:29.028098 10838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880383132139-392-0/minicluster-data/ts-1/data/instance:
uuid: "4c8a5ae182464ca194ad80e8cbcbfb79"
format_stamp: "Formatted at 2026-06-19 14:47:29 on dist-test-slave-2sb1"
server_key: "6141988f7e1257e6e3596de2e4bc33b3"
server_key_iv: "96b8cd64f079fb1bd9b58ed35eb4a76c"
server_key_version: "encryptionkey@0"
I20260619 14:47:29.028937 10838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880383132139-392-0/minicluster-data/ts-1/wal/instance:
uuid: "4c8a5ae182464ca194ad80e8cbcbfb79"
format_stamp: "Formatted at 2026-06-19 14:47:29 on dist-test-slave-2sb1"
server_key: "6141988f7e1257e6e3596de2e4bc33b3"
server_key_iv: "96b8cd64f079fb1bd9b58ed35eb4a76c"
server_key_version: "encryptionkey@0"
I20260619 14:47:29.033581 10838 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 14:47:29.037048 10853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:47:29.038679 10838 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:47:29.038883 10838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880383132139-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880383132139-392-0/minicluster-data/ts-1/wal
uuid: "4c8a5ae182464ca194ad80e8cbcbfb79"
format_stamp: "Formatted at 2026-06-19 14:47:29 on dist-test-slave-2sb1"
server_key: "6141988f7e1257e6e3596de2e4bc33b3"
server_key_iv: "96b8cd64f079fb1bd9b58ed35eb4a76c"
server_key_version: "encryptionkey@0"
I20260619 14:47:29.039021 10838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880383132139-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880383132139-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880383132139-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:47:29.049619 10838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:47:29.054407 10838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:47:29.054678 10838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:47:29.055738 10838 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:47:29.056972 10838 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:47:29.057062 10838 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:47:29.057132 10838 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:47:29.057156 10838 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:47:29.069082 10838 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33433
I20260619 14:47:29.069109 10966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33433 every 8 connection(s)
I20260619 14:47:29.070055 10838 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880383132139-392-0/minicluster-data/ts-1/data/info.pb
I20260619 14:47:29.071509 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 10838
I20260619 14:47:29.071624 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880383132139-392-0/minicluster-data/ts-1/wal/instance
I20260619 14:47:29.071957 392 external_mini_cluster.cc:1428] Setting key 4b6bb2a554387dccc97347c8ce961999
Jun 19 14:47:29 dist-test-slave-2sb1 krb5kdc[10597](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880449, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260619 14:47:29.089841 10681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:29.072417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55119 (local address 127.0.98.62:42553)
0619 14:47:29.072598 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:29.072604 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:29.073896 (+ 1292us) server_negotiation.cc:407] Connection header received
0619 14:47:29.075201 (+ 1305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:29.075204 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:29.075258 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:29.075338 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:29.077115 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:29.077707 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:29.079186 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:29.079417 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:29.079533 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:29.082865 (+ 3332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:47:29.082870 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:47:29.082872 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:47:29.082923 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:47:29.085585 (+ 2662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:29.086907 (+ 1322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:29.086913 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:29.086917 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:29.087222 (+ 305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:29.087866 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:29.087873 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:29.087878 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:29.088186 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:47:29.088311 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:29.089329 (+ 1018us) server_negotiation.cc:299] Negotiation successful
0619 14:47:29.089504 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 14:47:29.090989 10969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:29.073015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42553 (local address 127.0.98.2:55119)
0619 14:47:29.073751 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:29.073797 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:29.074978 (+ 1181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:29.075547 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:29.075560 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:29.076122 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:47:29.076952 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:29.076971 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:29.077874 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:29.077878 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:29.079034 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:29.079045 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:29.079585 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:29.079589 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:29.079788 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:29.080631 (+ 843us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:47:29.080737 (+ 106us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:47:29.082598 (+ 1861us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:47:29.085889 (+ 3291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:47:29.085895 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:47:29.085929 (+ 34us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:47:29.086650 (+ 721us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:47:29.087357 (+ 707us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:47:29.087360 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:47:29.087361 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:47:29.087706 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:47:29.088445 (+ 739us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:47:29.088451 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:47:29.088958 (+ 507us) client_negotiation.cc:769] Sending connection context
0619 14:47:29.089213 (+ 255us) client_negotiation.cc:240] Negotiation successful
0619 14:47:29.089725 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":255,"threads_started":1}
I20260619 14:47:29.092754 10967 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42553
I20260619 14:47:29.093183 10967 heartbeater.cc:464] Registering TS with master...
I20260619 14:47:29.093789 10967 heartbeater.cc:511] Master 127.0.98.62:42553 requested a full tablet report, sending...
I20260619 14:47:29.095180 10642 ts_manager.cc:194] Registered new tserver with Master: 4c8a5ae182464ca194ad80e8cbcbfb79 (127.0.98.2:33433)
I20260619 14:47:29.096065 10642 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55119
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880383132139-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880383132139-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 14:47:29.110373 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880383132139-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880383132139-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880383132139-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880383132139-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42553
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880383132139-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:47:29.234706 10972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:47:29.235011 10972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:47:29.235057 10972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:47:29.239950 10972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:47:29.240072 10972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:47:29.240232 10972 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 14:47:29.246805 10972 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880383132139-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880383132139-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880383132139-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880383132139-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42553
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10972
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880383132139-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:47:29.248023 10972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:47:29.249267 10972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:47:29.260982 10980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:47:29.261053 10978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:47:29.260943 10977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:47:29.260957 10972 server_base.cc:1034] running on GCE node
I20260619 14:47:29.261622 10972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:47:29.262357 10972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:47:29.263648 10972 hybrid_clock.cc:648] HybridClock initialized: now 1781880449263604 us; error 59 us; skew 500 ppm
Jun 19 14:47:29 dist-test-slave-2sb1 krb5kdc[10597](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880449, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:47:29.266896 10972 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 14:47:29.268121 10972 webserver.cc:460] Webserver started at http://127.0.98.3:35143/ using document root <none> and password file <none>
I20260619 14:47:29.268952 10972 fs_manager.cc:359] Metadata directory not provided
I20260619 14:47:29.269062 10972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:47:29.269372 10972 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:47:29.271883 10972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880383132139-392-0/minicluster-data/ts-2/data/instance:
uuid: "c209f352322344c8a96ff6ff16ec47b1"
format_stamp: "Formatted at 2026-06-19 14:47:29 on dist-test-slave-2sb1"
server_key: "be0ac490994ada8d21f3f6f86921e7aa"
server_key_iv: "cf51fd8cd12a8356e05de3f27654d289"
server_key_version: "encryptionkey@0"
I20260619 14:47:29.272637 10972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880383132139-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c209f352322344c8a96ff6ff16ec47b1"
format_stamp: "Formatted at 2026-06-19 14:47:29 on dist-test-slave-2sb1"
server_key: "be0ac490994ada8d21f3f6f86921e7aa"
server_key_iv: "cf51fd8cd12a8356e05de3f27654d289"
server_key_version: "encryptionkey@0"
I20260619 14:47:29.277448 10972 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 14:47:29.280584 10987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:47:29.282482 10972 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:47:29.282717 10972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880383132139-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880383132139-392-0/minicluster-data/ts-2/wal
uuid: "c209f352322344c8a96ff6ff16ec47b1"
format_stamp: "Formatted at 2026-06-19 14:47:29 on dist-test-slave-2sb1"
server_key: "be0ac490994ada8d21f3f6f86921e7aa"
server_key_iv: "cf51fd8cd12a8356e05de3f27654d289"
server_key_version: "encryptionkey@0"
I20260619 14:47:29.282985 10972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880383132139-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880383132139-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880383132139-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:47:29.302655 10972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:47:29.306311 10972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:47:29.306486 10972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:47:29.307039 10972 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:47:29.307924 10972 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:47:29.307973 10972 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:47:29.308015 10972 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:47:29.308032 10972 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:47:29.319401 10972 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44995
I20260619 14:47:29.319541 11100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44995 every 8 connection(s)
I20260619 14:47:29.320962 10972 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880383132139-392-0/minicluster-data/ts-2/data/info.pb
I20260619 14:47:29.329186 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 10972
I20260619 14:47:29.329342 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880383132139-392-0/minicluster-data/ts-2/wal/instance
I20260619 14:47:29.329664 392 external_mini_cluster.cc:1428] Setting key 9420eebab360f0a70bd9dcd2430bcd80
Jun 19 14:47:29 dist-test-slave-2sb1 krb5kdc[10597](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880449, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:47:29.339742 10681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:29.323276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39509 (local address 127.0.98.62:42553)
0619 14:47:29.323443 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:29.323449 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:29.325174 (+ 1725us) server_negotiation.cc:407] Connection header received
0619 14:47:29.326158 (+ 984us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:29.326161 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:29.326216 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:29.326306 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:29.327905 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:29.328761 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:29.330508 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:29.330968 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:29.331111 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:29.334591 (+ 3480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:47:29.334595 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:47:29.334597 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:47:29.334625 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:47:29.336512 (+ 1887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:29.337383 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:29.337390 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:29.337392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:29.337628 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:29.338224 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:29.338227 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:29.338229 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:29.338460 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:47:29.338578 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:29.339435 (+ 857us) server_negotiation.cc:299] Negotiation successful
0619 14:47:29.339586 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 14:47:29.340771 11103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:29.323797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42553 (local address 127.0.98.3:39509)
0619 14:47:29.324922 (+ 1125us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:29.325032 (+ 110us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:29.325877 (+ 845us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:29.326654 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:29.326662 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:29.327152 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:47:29.327716 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:29.327732 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:29.328927 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:29.328931 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:29.330339 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:29.330351 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:29.331215 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:29.331221 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:29.331483 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:29.332282 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:47:29.332305 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:47:29.334370 (+ 2065us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:47:29.336777 (+ 2407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:47:29.336784 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:47:29.336800 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:47:29.337195 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:47:29.337788 (+ 593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:47:29.337792 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:47:29.337793 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:47:29.338078 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:47:29.338691 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:47:29.338697 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:47:29.339132 (+ 435us) client_negotiation.cc:769] Sending connection context
0619 14:47:29.339328 (+ 196us) client_negotiation.cc:240] Negotiation successful
0619 14:47:29.339659 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":903,"thread_start_us":619,"threads_started":1}
I20260619 14:47:29.342319 11101 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42553
I20260619 14:47:29.342603 11101 heartbeater.cc:464] Registering TS with master...
I20260619 14:47:29.343220 11101 heartbeater.cc:511] Master 127.0.98.62:42553 requested a full tablet report, sending...
I20260619 14:47:29.344417 10642 ts_manager.cc:194] Registered new tserver with Master: c209f352322344c8a96ff6ff16ec47b1 (127.0.98.3:44995)
I20260619 14:47:29.345310 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:47:29.345336 10642 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:39509
I20260619 14:47:29.357372 10681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:29.348144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39926 (local address 127.0.98.62:42553)
0619 14:47:29.348319 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:29.348324 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:29.348471 (+ 147us) server_negotiation.cc:407] Connection header received
0619 14:47:29.348766 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:29.348769 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:29.348827 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:29.348919 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:47:29.349680 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:29.350212 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:29.351083 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:29.351321 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:29.351440 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:29.353414 (+ 1974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:47:29.353418 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:47:29.353421 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:47:29.353461 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:47:29.355420 (+ 1959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:29.355956 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:29.355960 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:29.355962 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:29.356072 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:47:29.356365 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:47:29.356368 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:47:29.356370 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:47:29.356624 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:47:29.356803 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:29.357072 (+ 269us) server_negotiation.cc:299] Negotiation successful
0619 14:47:29.357193 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260619 14:47:29.361099 10642 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39926:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:47:29.364423 10642 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:47:29.383780 11115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:29.374847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34983 (local address 127.0.0.1:46428)
0619 14:47:29.375768 (+ 921us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:29.375832 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:29.376060 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:29.377147 (+ 1087us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:29.377154 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:29.377188 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:47:29.377419 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:29.377427 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:29.381050 (+ 3623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:29.381054 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:29.382035 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:29.382045 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:29.383139 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:29.383143 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:29.383254 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:29.383338 (+ 84us) client_negotiation.cc:769] Sending connection context
0619 14:47:29.383463 (+ 125us) client_negotiation.cc:240] Negotiation successful
0619 14:47:29.383607 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":723,"mutex_wait_us":43,"spinlock_wait_cycles":22912,"thread_start_us":88,"threads_started":1}
I20260619 14:47:29.383850 11114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:29.375118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46428 (local address 127.0.98.1:34983)
0619 14:47:29.375481 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:29.375490 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:29.375938 (+ 448us) server_negotiation.cc:407] Connection header received
0619 14:47:29.376260 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:29.376266 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:29.376533 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:29.376694 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:47:29.379081 (+ 2387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:29.380798 (+ 1717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:29.382212 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:29.382961 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:29.383111 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:29.383270 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:29.383531 (+ 261us) server_negotiation.cc:299] Negotiation successful
0619 14:47:29.383657 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":97,"threads_started":1}
I20260619 14:47:29.384310 11116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:29.375813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33433 (local address 127.0.0.1:39126)
0619 14:47:29.376313 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:29.376329 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:29.376431 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:29.377142 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:29.377149 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:29.377196 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:47:29.377337 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:29.377343 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:29.379082 (+ 1739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:29.379086 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:29.381185 (+ 2099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:29.381196 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:29.383352 (+ 2156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:29.383356 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:29.383424 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:29.383484 (+ 60us) client_negotiation.cc:769] Sending connection context
0619 14:47:29.383980 (+ 496us) client_negotiation.cc:240] Negotiation successful
0619 14:47:29.384051 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"mutex_wait_us":300,"spinlock_wait_cycles":24320,"thread_start_us":101,"threads_started":1}
I20260619 14:47:29.384421 11117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:29.375980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39126 (local address 127.0.98.2:33433)
0619 14:47:29.376432 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:29.376445 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:29.376472 (+ 27us) server_negotiation.cc:407] Connection header received
0619 14:47:29.376555 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:29.376561 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:29.376834 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:29.377019 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:47:29.377473 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:29.378854 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:29.381339 (+ 2485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:29.382649 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:29.382777 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:29.382949 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:29.384161 (+ 1212us) server_negotiation.cc:299] Negotiation successful
0619 14:47:29.384267 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":110,"threads_started":1}
I20260619 14:47:29.384420 11113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:29.375027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38608 (local address 127.0.98.3:44995)
0619 14:47:29.375502 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:29.375516 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:29.375929 (+ 413us) server_negotiation.cc:407] Connection header received
0619 14:47:29.376606 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:29.376615 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:29.376851 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:29.376991 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:47:29.377560 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:29.378854 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:29.381410 (+ 2556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:29.382649 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:29.382778 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:29.382936 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:29.384073 (+ 1137us) server_negotiation.cc:299] Negotiation successful
0619 14:47:29.384266 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":152,"threads_started":1}
I20260619 14:47:29.384727 11112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:29.374847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44995 (local address 127.0.0.1:38608)
0619 14:47:29.375768 (+ 921us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:29.375832 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:29.376060 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:29.377195 (+ 1135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:29.377198 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:29.377219 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:47:29.377427 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:29.377434 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:29.379061 (+ 1627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:29.379110 (+ 49us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:29.381217 (+ 2107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:29.381229 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:29.382799 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:29.382803 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:29.383668 (+ 865us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:29.383693 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 14:47:29.383765 (+ 72us) client_negotiation.cc:240] Negotiation successful
0619 14:47:29.384449 (+ 684us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":759,"mutex_wait_us":707,"spinlock_wait_cycles":12160,"thread_start_us":128,"threads_started":1}
I20260619 14:47:29.387738 10767 tablet_service.cc:1467] Processing CreateTablet for tablet 60c194039f7b4093a118987ebf2a77d7 (DEFAULT_TABLE table=test-table [id=8108570715ef4ae58213ec1376785a31]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:47:29.388100 11034 tablet_service.cc:1467] Processing CreateTablet for tablet 60c194039f7b4093a118987ebf2a77d7 (DEFAULT_TABLE table=test-table [id=8108570715ef4ae58213ec1376785a31]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:47:29.388271 10901 tablet_service.cc:1467] Processing CreateTablet for tablet 60c194039f7b4093a118987ebf2a77d7 (DEFAULT_TABLE table=test-table [id=8108570715ef4ae58213ec1376785a31]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:47:29.389153 10767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60c194039f7b4093a118987ebf2a77d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:47:29.389173 11034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60c194039f7b4093a118987ebf2a77d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:47:29.389283 10901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60c194039f7b4093a118987ebf2a77d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:47:29.395869 11118 tablet_bootstrap.cc:492] T 60c194039f7b4093a118987ebf2a77d7 P c209f352322344c8a96ff6ff16ec47b1: Bootstrap starting.
I20260619 14:47:29.398365 11118 tablet_bootstrap.cc:654] T 60c194039f7b4093a118987ebf2a77d7 P c209f352322344c8a96ff6ff16ec47b1: Neither blocks nor log segments found. Creating new log.
I20260619 14:47:29.398887 11120 tablet_bootstrap.cc:492] T 60c194039f7b4093a118987ebf2a77d7 P 4c8a5ae182464ca194ad80e8cbcbfb79: Bootstrap starting.
I20260619 14:47:29.400104 11118 log.cc:824] T 60c194039f7b4093a118987ebf2a77d7 P c209f352322344c8a96ff6ff16ec47b1: Log is configured to *not* fsync() on all Append() calls
I20260619 14:47:29.400916 11119 tablet_bootstrap.cc:492] T 60c194039f7b4093a118987ebf2a77d7 P 8e94a8bcf0e84520aef549e11d72c6f3: Bootstrap starting.
I20260619 14:47:29.402383 11120 tablet_bootstrap.cc:654] T 60c194039f7b4093a118987ebf2a77d7 P 4c8a5ae182464ca194ad80e8cbcbfb79: Neither blocks nor log segments found. Creating new log.
I20260619 14:47:29.402829 11118 tablet_bootstrap.cc:492] T 60c194039f7b4093a118987ebf2a77d7 P c209f352322344c8a96ff6ff16ec47b1: No bootstrap required, opened a new log
I20260619 14:47:29.403038 11119 tablet_bootstrap.cc:654] T 60c194039f7b4093a118987ebf2a77d7 P 8e94a8bcf0e84520aef549e11d72c6f3: Neither blocks nor log segments found. Creating new log.
I20260619 14:47:29.403056 11118 ts_tablet_manager.cc:1397] T 60c194039f7b4093a118987ebf2a77d7 P c209f352322344c8a96ff6ff16ec47b1: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260619 14:47:29.403494 11120 log.cc:824] T 60c194039f7b4093a118987ebf2a77d7 P 4c8a5ae182464ca194ad80e8cbcbfb79: Log is configured to *not* fsync() on all Append() calls
I20260619 14:47:29.403795 11119 log.cc:824] T 60c194039f7b4093a118987ebf2a77d7 P 8e94a8bcf0e84520aef549e11d72c6f3: Log is configured to *not* fsync() on all Append() calls
I20260619 14:47:29.405646 11120 tablet_bootstrap.cc:492] T 60c194039f7b4093a118987ebf2a77d7 P 4c8a5ae182464ca194ad80e8cbcbfb79: No bootstrap required, opened a new log
I20260619 14:47:29.405872 11119 tablet_bootstrap.cc:492] T 60c194039f7b4093a118987ebf2a77d7 P 8e94a8bcf0e84520aef549e11d72c6f3: No bootstrap required, opened a new log
I20260619 14:47:29.405898 11120 ts_tablet_manager.cc:1397] T 60c194039f7b4093a118987ebf2a77d7 P 4c8a5ae182464ca194ad80e8cbcbfb79: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260619 14:47:29.406059 11119 ts_tablet_manager.cc:1397] T 60c194039f7b4093a118987ebf2a77d7 P 8e94a8bcf0e84520aef549e11d72c6f3: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260619 14:47:29.407437 11118 raft_consensus.cc:348] T 60c194039f7b4093a118987ebf2a77d7 P c209f352322344c8a96ff6ff16ec47b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c209f352322344c8a96ff6ff16ec47b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44995 } } peers { permanent_uuid: "8e94a8bcf0e84520aef549e11d72c6f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34983 } } peers { permanent_uuid: "4c8a5ae182464ca194ad80e8cbcbfb79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33433 } }
I20260619 14:47:29.407786 11118 raft_consensus.cc:374] T 60c194039f7b4093a118987ebf2a77d7 P c209f352322344c8a96ff6ff16ec47b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:47:29.407856 11118 raft_consensus.cc:729] T 60c194039f7b4093a118987ebf2a77d7 P c209f352322344c8a96ff6ff16ec47b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c209f352322344c8a96ff6ff16ec47b1, State: Initialized, Role: FOLLOWER
I20260619 14:47:29.408407 11118 consensus_queue.cc:260] T 60c194039f7b4093a118987ebf2a77d7 P c209f352322344c8a96ff6ff16ec47b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c209f352322344c8a96ff6ff16ec47b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44995 } } peers { permanent_uuid: "8e94a8bcf0e84520aef549e11d72c6f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34983 } } peers { permanent_uuid: "4c8a5ae182464ca194ad80e8cbcbfb79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33433 } }
I20260619 14:47:29.409240 11101 heartbeater.cc:503] Master 127.0.98.62:42553 was elected leader, sending a full tablet report...
I20260619 14:47:29.409590 11118 ts_tablet_manager.cc:1428] T 60c194039f7b4093a118987ebf2a77d7 P c209f352322344c8a96ff6ff16ec47b1: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260619 14:47:29.409883 11120 raft_consensus.cc:348] T 60c194039f7b4093a118987ebf2a77d7 P 4c8a5ae182464ca194ad80e8cbcbfb79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c209f352322344c8a96ff6ff16ec47b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44995 } } peers { permanent_uuid: "8e94a8bcf0e84520aef549e11d72c6f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34983 } } peers { permanent_uuid: "4c8a5ae182464ca194ad80e8cbcbfb79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33433 } }
I20260619 14:47:29.410166 11120 raft_consensus.cc:374] T 60c194039f7b4093a118987ebf2a77d7 P 4c8a5ae182464ca194ad80e8cbcbfb79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:47:29.410228 11120 raft_consensus.cc:729] T 60c194039f7b4093a118987ebf2a77d7 P 4c8a5ae182464ca194ad80e8cbcbfb79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c8a5ae182464ca194ad80e8cbcbfb79, State: Initialized, Role: FOLLOWER
I20260619 14:47:29.410758 11120 consensus_queue.cc:260] T 60c194039f7b4093a118987ebf2a77d7 P 4c8a5ae182464ca194ad80e8cbcbfb79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c209f352322344c8a96ff6ff16ec47b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44995 } } peers { permanent_uuid: "8e94a8bcf0e84520aef549e11d72c6f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34983 } } peers { permanent_uuid: "4c8a5ae182464ca194ad80e8cbcbfb79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33433 } }
I20260619 14:47:29.411479 10967 heartbeater.cc:503] Master 127.0.98.62:42553 was elected leader, sending a full tablet report...
I20260619 14:47:29.411813 11119 raft_consensus.cc:348] T 60c194039f7b4093a118987ebf2a77d7 P 8e94a8bcf0e84520aef549e11d72c6f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c209f352322344c8a96ff6ff16ec47b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44995 } } peers { permanent_uuid: "8e94a8bcf0e84520aef549e11d72c6f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34983 } } peers { permanent_uuid: "4c8a5ae182464ca194ad80e8cbcbfb79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33433 } }
I20260619 14:47:29.412029 11120 ts_tablet_manager.cc:1428] T 60c194039f7b4093a118987ebf2a77d7 P 4c8a5ae182464ca194ad80e8cbcbfb79: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 14:47:29.412087 11119 raft_consensus.cc:374] T 60c194039f7b4093a118987ebf2a77d7 P 8e94a8bcf0e84520aef549e11d72c6f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:47:29.412142 11119 raft_consensus.cc:729] T 60c194039f7b4093a118987ebf2a77d7 P 8e94a8bcf0e84520aef549e11d72c6f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e94a8bcf0e84520aef549e11d72c6f3, State: Initialized, Role: FOLLOWER
I20260619 14:47:29.412642 11119 consensus_queue.cc:260] T 60c194039f7b4093a118987ebf2a77d7 P 8e94a8bcf0e84520aef549e11d72c6f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c209f352322344c8a96ff6ff16ec47b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44995 } } peers { permanent_uuid: "8e94a8bcf0e84520aef549e11d72c6f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34983 } } peers { permanent_uuid: "4c8a5ae182464ca194ad80e8cbcbfb79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33433 } }
I20260619 14:47:29.413417 10833 heartbeater.cc:503] Master 127.0.98.62:42553 was elected leader, sending a full tablet report...
I20260619 14:47:29.413707 11119 ts_tablet_manager.cc:1428] T 60c194039f7b4093a118987ebf2a77d7 P 8e94a8bcf0e84520aef549e11d72c6f3: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260619 14:47:29.517637 11125 raft_consensus.cc:482] T 60c194039f7b4093a118987ebf2a77d7 P 4c8a5ae182464ca194ad80e8cbcbfb79 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:47:29.518085 11125 raft_consensus.cc:504] T 60c194039f7b4093a118987ebf2a77d7 P 4c8a5ae182464ca194ad80e8cbcbfb79 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c209f352322344c8a96ff6ff16ec47b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44995 } } peers { permanent_uuid: "8e94a8bcf0e84520aef549e11d72c6f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34983 } } peers { permanent_uuid: "4c8a5ae182464ca194ad80e8cbcbfb79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33433 } }
I20260619 14:47:29.519344 11125 leader_election.cc:288] T 60c194039f7b4093a118987ebf2a77d7 P 4c8a5ae182464ca194ad80e8cbcbfb79 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c209f352322344c8a96ff6ff16ec47b1 (127.0.98.3:44995), 8e94a8bcf0e84520aef549e11d72c6f3 (127.0.98.1:34983)
I20260619 14:47:29.525543 11114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:29.520115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47311 (local address 127.0.98.1:34983)
0619 14:47:29.520262 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:29.520267 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:29.520559 (+ 292us) server_negotiation.cc:407] Connection header received
0619 14:47:29.520614 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:29.520616 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:29.520661 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:29.520846 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:47:29.521093 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:29.521942 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:29.523942 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:29.524717 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:29.524848 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:29.524941 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:29.525301 (+ 360us) server_negotiation.cc:299] Negotiation successful
0619 14:47:29.525390 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 14:47:29.525586 11113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:29.519702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37865 (local address 127.0.98.3:44995)
0619 14:47:29.519846 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:29.519853 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:29.519938 (+ 85us) server_negotiation.cc:407] Connection header received
0619 14:47:29.520202 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:29.520206 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:29.520276 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:29.520365 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:47:29.520807 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:29.521718 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:29.523942 (+ 2224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:29.524832 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:29.524936 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:29.525004 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:29.525301 (+ 297us) server_negotiation.cc:299] Negotiation successful
0619 14:47:29.525390 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 14:47:29.525566 10969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:29.519576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44995 (local address 127.0.98.2:37865)
0619 14:47:29.519845 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:29.519863 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:29.519996 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:29.520411 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:29.520414 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:29.520436 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:47:29.520606 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:29.520615 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:29.521921 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:29.521924 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:29.523570 (+ 1646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:29.523580 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:29.524961 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:29.524964 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:29.525090 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:29.525146 (+ 56us) client_negotiation.cc:769] Sending connection context
0619 14:47:29.525246 (+ 100us) client_negotiation.cc:240] Negotiation successful
0619 14:47:29.525345 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"mutex_wait_us":190}
I20260619 14:47:29.525564 11127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:29.519930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34983 (local address 127.0.98.2:47311)
0619 14:47:29.520417 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:29.520430 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:47:29.520528 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:47:29.520818 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:47:29.520820 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:47:29.520840 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:47:29.520974 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:29.520979 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:29.522077 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:29.522082 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:29.523566 (+ 1484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:47:29.523580 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:47:29.524941 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:47:29.524955 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:47:29.525086 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:29.525156 (+ 70us) client_negotiation.cc:769] Sending connection context
0619 14:47:29.525246 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 14:47:29.525345 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"mutex_wait_us":111,"thread_start_us":188,"threads_started":1}
I20260619 14:47:29.526453 11055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60c194039f7b4093a118987ebf2a77d7" candidate_uuid: "4c8a5ae182464ca194ad80e8cbcbfb79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c209f352322344c8a96ff6ff16ec47b1" is_pre_election: true
I20260619 14:47:29.526757 10787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60c194039f7b4093a118987ebf2a77d7" candidate_uuid: "4c8a5ae182464ca194ad80e8cbcbfb79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e94a8bcf0e84520aef549e11d72c6f3" is_pre_election: true
I20260619 14:47:29.526834 11055 raft_consensus.cc:2454] T 60c194039f7b4093a118987ebf2a77d7 P c209f352322344c8a96ff6ff16ec47b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8a5ae182464ca194ad80e8cbcbfb79 in term 0.
I20260619 14:47:29.527143 10787 raft_consensus.cc:2454] T 60c194039f7b4093a118987ebf2a77d7 P 8e94a8bcf0e84520aef549e11d72c6f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c8a5ae182464ca194ad80e8cbcbfb79 in term 0.
I20260619 14:47:29.527519 10854 leader_election.cc:302] T 60c194039f7b4093a118987ebf2a77d7 P 4c8a5ae182464ca194ad80e8cbcbfb79 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8a5ae182464ca194ad80e8cbcbfb79, c209f352322344c8a96ff6ff16ec47b1; no voters:
I20260619 14:47:29.527863 11125 raft_consensus.cc:2781] T 60c194039f7b4093a118987ebf2a77d7 P 4c8a5ae182464ca194ad80e8cbcbfb79 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:47:29.527959 11125 raft_consensus.cc:482] T 60c194039f7b4093a118987ebf2a77d7 P 4c8a5ae182464ca194ad80e8cbcbfb79 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:47:29.528004 11125 raft_consensus.cc:3037] T 60c194039f7b4093a118987ebf2a77d7 P 4c8a5ae182464ca194ad80e8cbcbfb79 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:47:29.529137 11125 raft_consensus.cc:504] T 60c194039f7b4093a118987ebf2a77d7 P 4c8a5ae182464ca194ad80e8cbcbfb79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c209f352322344c8a96ff6ff16ec47b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44995 } } peers { permanent_uuid: "8e94a8bcf0e84520aef549e11d72c6f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34983 } } peers { permanent_uuid: "4c8a5ae182464ca194ad80e8cbcbfb79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33433 } }
I20260619 14:47:29.529569 11125 leader_election.cc:288] T 60c194039f7b4093a118987ebf2a77d7 P 4c8a5ae182464ca194ad80e8cbcbfb79 [CANDIDATE]: Term 1 election: Requested vote from peers c209f352322344c8a96ff6ff16ec47b1 (127.0.98.3:44995), 8e94a8bcf0e84520aef549e11d72c6f3 (127.0.98.1:34983)
I20260619 14:47:29.529987 11055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60c194039f7b4093a118987ebf2a77d7" candidate_uuid: "4c8a5ae182464ca194ad80e8cbcbfb79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c209f352322344c8a96ff6ff16ec47b1"
I20260619 14:47:29.530119 11055 raft_consensus.cc:3037] T 60c194039f7b4093a118987ebf2a77d7 P c209f352322344c8a96ff6ff16ec47b1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:47:29.530113 10787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60c194039f7b4093a118987ebf2a77d7" candidate_uuid: "4c8a5ae182464ca194ad80e8cbcbfb79" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e94a8bcf0e84520aef549e11d72c6f3"
I20260619 14:47:29.530244 10787 raft_consensus.cc:3037] T 60c194039f7b4093a118987ebf2a77d7 P 8e94a8bcf0e84520aef549e11d72c6f3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:47:29.531386 10787 raft_consensus.cc:2454] T 60c194039f7b4093a118987ebf2a77d7 P 8e94a8bcf0e84520aef549e11d72c6f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c8a5ae182464ca194ad80e8cbcbfb79 in term 1.
I20260619 14:47:29.531493 11055 raft_consensus.cc:2454] T 60c194039f7b4093a118987ebf2a77d7 P c209f352322344c8a96ff6ff16ec47b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c8a5ae182464ca194ad80e8cbcbfb79 in term 1.
I20260619 14:47:29.531785 10857 leader_election.cc:302] T 60c194039f7b4093a118987ebf2a77d7 P 4c8a5ae182464ca194ad80e8cbcbfb79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c8a5ae182464ca194ad80e8cbcbfb79, 8e94a8bcf0e84520aef549e11d72c6f3; no voters:
I20260619 14:47:29.531971 11125 raft_consensus.cc:2781] T 60c194039f7b4093a118987ebf2a77d7 P 4c8a5ae182464ca194ad80e8cbcbfb79 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:47:29.532375 11125 raft_consensus.cc:686] T 60c194039f7b4093a118987ebf2a77d7 P 4c8a5ae182464ca194ad80e8cbcbfb79 [term 1 LEADER]: Becoming Leader. State: Replica: 4c8a5ae182464ca194ad80e8cbcbfb79, State: Running, Role: LEADER
I20260619 14:47:29.532788 11125 consensus_queue.cc:237] T 60c194039f7b4093a118987ebf2a77d7 P 4c8a5ae182464ca194ad80e8cbcbfb79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c209f352322344c8a96ff6ff16ec47b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44995 } } peers { permanent_uuid: "8e94a8bcf0e84520aef549e11d72c6f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34983 } } peers { permanent_uuid: "4c8a5ae182464ca194ad80e8cbcbfb79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33433 } }
I20260619 14:47:29.535967 10642 catalog_manager.cc:5495] T 60c194039f7b4093a118987ebf2a77d7 P 4c8a5ae182464ca194ad80e8cbcbfb79 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4c8a5ae182464ca194ad80e8cbcbfb79 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "4c8a5ae182464ca194ad80e8cbcbfb79" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c209f352322344c8a96ff6ff16ec47b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44995 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e94a8bcf0e84520aef549e11d72c6f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34983 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c8a5ae182464ca194ad80e8cbcbfb79" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33433 } health_report { overall_health: HEALTHY } } }
W20260619 14:47:29.571303 10968 tablet.cc:2369] T 60c194039f7b4093a118987ebf2a77d7 P 4c8a5ae182464ca194ad80e8cbcbfb79: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 14:47:29.572178 11102 tablet.cc:2369] T 60c194039f7b4093a118987ebf2a77d7 P c209f352322344c8a96ff6ff16ec47b1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:47:29.578363 11117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:29.573699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39136 (local address 127.0.98.2:33433)
0619 14:47:29.573838 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 14:47:29.573841 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:47:29.573856 (+ 15us) server_negotiation.cc:407] Connection header received
0619 14:47:29.573965 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:47:29.573968 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:47:29.574036 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:47:29.574145 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:47:29.574469 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:29.575035 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:29.576139 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:47:29.576399 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:47:29.576533 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:47:29.577169 (+ 636us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:47:29.577671 (+ 502us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:47:29.577782 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 14:47:29.578025 (+ 243us) server_negotiation.cc:299] Negotiation successful
0619 14:47:29.578085 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260619 14:47:29.578922 10834 tablet.cc:2369] T 60c194039f7b4093a118987ebf2a77d7 P 8e94a8bcf0e84520aef549e11d72c6f3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:47:29.585405 10787 raft_consensus.cc:1261] T 60c194039f7b4093a118987ebf2a77d7 P 8e94a8bcf0e84520aef549e11d72c6f3 [term 1 FOLLOWER]: Refusing update from remote peer 4c8a5ae182464ca194ad80e8cbcbfb79: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:47:29.585461 11055 raft_consensus.cc:1261] T 60c194039f7b4093a118987ebf2a77d7 P c209f352322344c8a96ff6ff16ec47b1 [term 1 FOLLOWER]: Refusing update from remote peer 4c8a5ae182464ca194ad80e8cbcbfb79: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:47:29.586303 11125 consensus_queue.cc:1035] T 60c194039f7b4093a118987ebf2a77d7 P 4c8a5ae182464ca194ad80e8cbcbfb79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c209f352322344c8a96ff6ff16ec47b1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:47:29.586577 11128 consensus_queue.cc:1035] T 60c194039f7b4093a118987ebf2a77d7 P 4c8a5ae182464ca194ad80e8cbcbfb79 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e94a8bcf0e84520aef549e11d72c6f3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:47:29.596477 11132 mvcc.cc:205] Tried to move back new op lower bound from 7298582321493250048 to 7298582321289084928. Current Snapshot: MvccSnapshot[applied={T|T < 7298582321493250048}]
I20260619 14:47:29.597385 11131 mvcc.cc:205] Tried to move back new op lower bound from 7298582321493250048 to 7298582321289084928. Current Snapshot: MvccSnapshot[applied={T|T < 7298582321493250048}]
I20260619 14:47:29.609951 10642 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39926:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:47:29.610168 10642 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39926:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:47:29.613597 10642 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ecde9ecb467c44d8ac03c9ad2971b088: Sending DeleteTablet for 3 replicas of tablet 60c194039f7b4093a118987ebf2a77d7
I20260619 14:47:29.614657 11034 tablet_service.cc:1514] Processing DeleteTablet for tablet 60c194039f7b4093a118987ebf2a77d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:47:29 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38608
I20260619 14:47:29.614730 10767 tablet_service.cc:1514] Processing DeleteTablet for tablet 60c194039f7b4093a118987ebf2a77d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:47:29 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46428
I20260619 14:47:29.614776 10901 tablet_service.cc:1514] Processing DeleteTablet for tablet 60c194039f7b4093a118987ebf2a77d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:47:29 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39126
I20260619 14:47:29.615280 11142 tablet_replica.cc:330] T 60c194039f7b4093a118987ebf2a77d7 P 8e94a8bcf0e84520aef549e11d72c6f3: stopping tablet replica
I20260619 14:47:29.615513 11141 tablet_replica.cc:330] T 60c194039f7b4093a118987ebf2a77d7 P 4c8a5ae182464ca194ad80e8cbcbfb79: stopping tablet replica
I20260619 14:47:29.615691 11142 raft_consensus.cc:2229] T 60c194039f7b4093a118987ebf2a77d7 P 8e94a8bcf0e84520aef549e11d72c6f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:47:29.615980 11141 raft_consensus.cc:2229] T 60c194039f7b4093a118987ebf2a77d7 P 4c8a5ae182464ca194ad80e8cbcbfb79 [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:47:29.616029 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 10704
I20260619 14:47:29.616072 11142 raft_consensus.cc:2258] T 60c194039f7b4093a118987ebf2a77d7 P 8e94a8bcf0e84520aef549e11d72c6f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:47:29.616839 11140 tablet_replica.cc:330] T 60c194039f7b4093a118987ebf2a77d7 P c209f352322344c8a96ff6ff16ec47b1: stopping tablet replica
I20260619 14:47:29.617084 11141 raft_consensus.cc:2258] T 60c194039f7b4093a118987ebf2a77d7 P 4c8a5ae182464ca194ad80e8cbcbfb79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:47:29.617182 11140 raft_consensus.cc:2229] T 60c194039f7b4093a118987ebf2a77d7 P c209f352322344c8a96ff6ff16ec47b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:47:29.617756 11140 raft_consensus.cc:2258] T 60c194039f7b4093a118987ebf2a77d7 P c209f352322344c8a96ff6ff16ec47b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:47:29.619401 11141 ts_tablet_manager.cc:1905] T 60c194039f7b4093a118987ebf2a77d7 P 4c8a5ae182464ca194ad80e8cbcbfb79: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:47:29.619401 11140 ts_tablet_manager.cc:1905] T 60c194039f7b4093a118987ebf2a77d7 P c209f352322344c8a96ff6ff16ec47b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:47:29.622434 11140 ts_tablet_manager.cc:1918] T 60c194039f7b4093a118987ebf2a77d7 P c209f352322344c8a96ff6ff16ec47b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:47:29.622435 11141 ts_tablet_manager.cc:1918] T 60c194039f7b4093a118987ebf2a77d7 P 4c8a5ae182464ca194ad80e8cbcbfb79: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:47:29.622561 11140 log.cc:1196] T 60c194039f7b4093a118987ebf2a77d7 P c209f352322344c8a96ff6ff16ec47b1: Deleting WAL directory at /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880383132139-392-0/minicluster-data/ts-2/wal/wals/60c194039f7b4093a118987ebf2a77d7
I20260619 14:47:29.622561 11141 log.cc:1196] T 60c194039f7b4093a118987ebf2a77d7 P 4c8a5ae182464ca194ad80e8cbcbfb79: Deleting WAL directory at /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781880383132139-392-0/minicluster-data/ts-1/wal/wals/60c194039f7b4093a118987ebf2a77d7
I20260619 14:47:29.622893 11141 ts_tablet_manager.cc:1939] T 60c194039f7b4093a118987ebf2a77d7 P 4c8a5ae182464ca194ad80e8cbcbfb79: Deleting consensus metadata
I20260619 14:47:29.622895 11140 ts_tablet_manager.cc:1939] T 60c194039f7b4093a118987ebf2a77d7 P c209f352322344c8a96ff6ff16ec47b1: Deleting consensus metadata
I20260619 14:47:29.624218 10626 catalog_manager.cc:4841] TS 4c8a5ae182464ca194ad80e8cbcbfb79 (127.0.98.2:33433): tablet 60c194039f7b4093a118987ebf2a77d7 (table test-table [id=8108570715ef4ae58213ec1376785a31]) successfully deleted
I20260619 14:47:29.624497 10626 catalog_manager.cc:4841] TS c209f352322344c8a96ff6ff16ec47b1 (127.0.98.3:44995): tablet 60c194039f7b4093a118987ebf2a77d7 (table test-table [id=8108570715ef4ae58213ec1376785a31]) successfully deleted
W20260619 14:47:29.625468 10629 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34983 (error 108)
I20260619 14:47:29.626395 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 10838
I20260619 14:47:29.626641 11112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:47:29.626298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34983 (local address 127.0.0.1:46434)
0619 14:47:29.626476 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0619 14:47:29.626555 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260619 14:47:29.628196 10629 catalog_manager.cc:4584] TS 8e94a8bcf0e84520aef549e11d72c6f3 (127.0.98.1:34983): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 60c194039f7b4093a118987ebf2a77d7: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34983: connect: Connection refused (error 111)
I20260619 14:47:29.632692 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 10972
I20260619 14:47:29.639189 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 10610
2026-06-19T14:47:29Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2602 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:47:29 dist-test-slave-2sb1 krb5kdc[11146](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:47:29 dist-test-slave-2sb1 krb5kdc[11146](info): set up 2 sockets
Jun 19 14:47:29 dist-test-slave-2sb1 krb5kdc[11146](info): commencing operation
krb5kdc: starting...
W20260619 14:47:30.762104 392 mini_kdc.cc:121] Time spent starting KDC: real 1.100s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/krb5kdc/test-user.keytab.
Jun 19 14:47:30 dist-test-slave-2sb1 krb5kdc[11146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880450, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:47:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:47:30Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260619 14:47:30.964910 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskYPr0M2/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/minicluster-data/postgres -l logfile start
2026-06-19 14:47:34.478 UTC [11176] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 14:47:34.478 UTC [11176] LOG: listening on IPv4 address "127.0.98.20", port 38135
2026-06-19 14:47:34.484 UTC [11176] LOG: listening on Unix socket "/tmp/.s.PGSQL.38135"
2026-06-19 14:47:34.511 UTC [11179] LOG: database system was shut down at 2026-06-19 14:47:32 UTC
2026-06-19 14:47:34.518 UTC [11176] LOG: database system is ready to accept connections
I20260619 14:47:35.566628 392 mini_postgres.cc:96] Postgres bound to 38135
2026-06-19 14:47:35.573 UTC [11187] FATAL: database "slave" does not exist
127.0.98.20:38135 - accepting connections
I20260619 14:47:35.574556 392 mini_ranger.cc:161] Starting Ranger...
I20260619 14:47:35.592862 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 14:47:36.526223 392 mini_ranger.cc:87] Created ranger Postgres database
I20260619 14:47:36.526342 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/minicluster-data/ranger-admin
2026-06-19 14:47:36,866 [I] DB FLAVOR :POSTGRES
2026-06-19 14:47:36,866 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 14:47:36,866 [I] Checking connection..
2026-06-19 14:47:36,867 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYPr0M2/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYPr0M2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38135/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 14:47:37,196 [I] Checking connection passed.
2026-06-19 14:47:37,197 [I] --------- Verifying version history table ---------
2026-06-19 14:47:37,197 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYPr0M2/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYPr0M2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38135/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 14:47:37,471 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 14:47:37,471 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYPr0M2/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYPr0M2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38135/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 14:47:37,766 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 14:47:37,767 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 14:47:37,767 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYPr0M2/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYPr0M2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38135/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskYPr0M2/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 14:47:38.053 UTC [11251] WARNING: there is no transaction in progress
2026-06-19 14:47:38,061 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 14:47:38,061 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYPr0M2/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYPr0M2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38135/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 14:47:38,390 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 14:47:38,391 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 14:47:38,391 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYPr0M2/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYPr0M2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38135/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 14:47:38,680 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYPr0M2/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYPr0M2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38135/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 14:47:38,957 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYPr0M2/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYPr0M2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38135/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-2sb1.c.gcp-upstream.internal','N') ;"
2026-06-19 14:47:39,259 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYPr0M2/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYPr0M2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38135/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 14:47:39,536 [I] Table x_portal_user does not exist in database ranger
2026-06-19 14:47:39,537 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYPr0M2/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYPr0M2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38135/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 14:47:39,807 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 14:47:39,807 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYPr0M2/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYPr0M2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38135/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 14:47:40,098 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYPr0M2/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYPr0M2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38135/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 14:47:40,399 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 14:47:40,399 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYPr0M2/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYPr0M2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38135/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskYPr0M2/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 14:47:41.858 UTC [11359] WARNING: there is no transaction in progress
2026-06-19 14:47:41.883 UTC [11359] WARNING: there is no transaction in progress
2026-06-19 14:47:41.909 UTC [11359] WARNING: there is no transaction in progress
2026-06-19 14:47:41.934 UTC [11359] WARNING: there is no transaction in progress
2026-06-19 14:47:41.961 UTC [11359] WARNING: there is no transaction in progress
2026-06-19 14:47:41.990 UTC [11359] WARNING: there is no transaction in progress
2026-06-19 14:47:42.072 UTC [11359] WARNING: there is no transaction in progress
2026-06-19 14:47:42.085 UTC [11359] WARNING: there is no transaction in progress
2026-06-19 14:47:42.110 UTC [11359] WARNING: there is no transaction in progress
2026-06-19 14:47:42.126 UTC [11359] WARNING: there is no transaction in progress
2026-06-19 14:47:42.143 UTC [11359] WARNING: there is no transaction in progress
2026-06-19 14:47:42.167 UTC [11359] WARNING: there is no transaction in progress
2026-06-19 14:47:42.185 UTC [11359] WARNING: there is no transaction in progress
2026-06-19 14:47:42.196 UTC [11359] WARNING: there is no transaction in progress
2026-06-19 14:47:42.208 UTC [11359] WARNING: there is no transaction in progress
2026-06-19 14:47:42.222 UTC [11359] WARNING: there is no transaction in progress
2026-06-19 14:47:43,427 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 14:47:43,428 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYPr0M2/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYPr0M2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38135/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 14:47:43,718 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 14:47:43,718 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYPr0M2/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYPr0M2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38135/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 14:47:44,010 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 14:47:44,011 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYPr0M2/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYPr0M2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38135/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 14:47:44,296 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 14:47:44,297 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYPr0M2/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYPr0M2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38135/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 14:47:44,567 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYPr0M2/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYPr0M2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38135/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 14:47:44,838 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYPr0M2/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYPr0M2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38135/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-2sb1.c.gcp-upstream.internal';"
2026-06-19 14:47:45,121 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 14:47:45,122 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYPr0M2/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYPr0M2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38135/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 14:47:45,436 [I] DB_PATCHES have already been applied
I20260619 14:47:45.442616 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskYPr0M2/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskYPr0M2/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskYPr0M2/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYPr0M2/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 14:47:45.442708 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260619 14:47:45.446555 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:54435
Jun 19, 2026 2:47:45 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 2:47:45 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 2:47:45 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 2:47:45 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 2:47:45 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 2:47:46 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 2:47:46 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 14:47:46.688834 392 mini_ranger.cc:160] Time spent starting Ranger: real 11.114s user 0.001s sys 0.011s
Jun 19, 2026 2:48:11 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 2:48:11 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Jun 19, 2026 2:48:11 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 2:48:11 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 2:48:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 2:48:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 2:48:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 2:48:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 2:48:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 2:48:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 2:48:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 2:48:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 2:48:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 2:48:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 2:48:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 2:48:11 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 2:48:11 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 2:48:12 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 14:48:13.034101 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 14:48:13 dist-test-slave-2sb1 krb5kdc[11146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880493, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 14:48:13.163229 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 14:48:13.176728 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 14:48:14.599963 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 14:48:14.600071 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/minicluster-data/ranger-kms
I20260619 14:48:14.600253 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:38135
2026-06-19 14:48:14,812 [I] DB FLAVOR :POSTGRES
2026-06-19 14:48:14,813 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 14:48:14,813 [I] Checking connection
2026-06-19 14:48:14,813 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYPr0M2/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38135/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 14:48:15,113 [I] connection success
2026-06-19 14:48:15,114 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 14:48:15,114 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 14:48:15,114 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYPr0M2/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38135/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 14:48:15,413 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 14:48:15,413 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 14:48:15,413 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 14:48:15,414 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYPr0M2/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:38135/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 14:48:15,740 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 14:48:16.370849 392 mini_ranger_kms.cc:310] Created kms service
I20260619 14:48:16.625813 392 mini_ranger_kms.cc:326] Created kudu user
I20260619 14:48:16.689996 392 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 14:48:16 dist-test-slave-2sb1 krb5kdc[11146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880493, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 14:48:17.216171 392 mini_ranger_kms.cc:384] Added ranger policy
I20260619 14:48:17.216416 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskYPr0M2/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYPr0M2/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskYPr0M2/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 14:48:17.216442 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260619 14:48:17.220541 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11540
I20260619 14:48:17.220739 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:51683
Jun 19, 2026 2:48:18 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 2:48:18 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 2:48:18 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 2:48:18 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 2:48:18 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 2:48:18 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Jun 19 14:48:19 dist-test-slave-2sb1 krb5kdc[11146](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781880499, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:48:19.722365 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.559s user 0.000s sys 0.197s
I20260619 14:48:19.722506 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 14:48:19.722606 392 mini_ranger_kms.cc:401] 127.0.98.20:51683/kms/v1/keys
Jun 19 14:48:20 dist-test-slave-2sb1 krb5kdc[11146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880499, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 19 14:48:20 dist-test-slave-2sb1 krb5kdc[11146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880499, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 19 14:48:23 dist-test-slave-2sb1 krb5kdc[11146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880503, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 14:48:23.061494 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35321
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35321
--ranger_config_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:51683/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:48:23.198508 11590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:48:23.198851 11590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:48:23.198899 11590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:48:23.203715 11590 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:48:23.203836 11590 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:48:23.203857 11590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:48:23.203874 11590 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:48:23.203890 11590 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:48:23.210322 11590 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:51683/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35321
--ranger_config_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35321
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11590
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:48:23.211536 11590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:48:23.212801 11590 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:48:23.223487 11595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:48:23.223642 11598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:48:23.223477 11596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:48:23.223824 11590 server_base.cc:1034] running on GCE node
I20260619 14:48:23.224701 11590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:48:23.226540 11590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:48:23.227771 11590 hybrid_clock.cc:648] HybridClock initialized: now 1781880503227740 us; error 58 us; skew 500 ppm
Jun 19 14:48:23 dist-test-slave-2sb1 krb5kdc[11146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880503, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:48:23.237545 11590 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 14:48:23.238993 11590 webserver.cc:460] Webserver started at http://127.0.98.62:36285/ using document root <none> and password file <none>
I20260619 14:48:23.239688 11590 fs_manager.cc:359] Metadata directory not provided
I20260619 14:48:23.239799 11590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:48:23.240060 11590 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 14:48:23 dist-test-slave-2sb1 krb5kdc[11146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880503, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 14:48:23.311906 11590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/minicluster-data/master-0/data/instance:
uuid: "da0bdfb0db164fda9731b9c2086cbd27"
format_stamp: "Formatted at 2026-06-19 14:48:23 on dist-test-slave-2sb1"
server_key: "fcc4428896b7e83c4c65062f5ba39f4e"
server_key_iv: "acb0043d3a195e4785d2f30bafe4d5df"
server_key_version: "kuduclusterkey@0"
I20260619 14:48:23.313077 11590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/minicluster-data/master-0/wal/instance:
uuid: "da0bdfb0db164fda9731b9c2086cbd27"
format_stamp: "Formatted at 2026-06-19 14:48:23 on dist-test-slave-2sb1"
server_key: "fcc4428896b7e83c4c65062f5ba39f4e"
server_key_iv: "acb0043d3a195e4785d2f30bafe4d5df"
server_key_version: "kuduclusterkey@0"
I20260619 14:48:23.319095 11590 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260619 14:48:23.340816 11605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:48:23.342573 11590 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 14:48:23.342746 11590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/minicluster-data/master-0/wal
uuid: "da0bdfb0db164fda9731b9c2086cbd27"
format_stamp: "Formatted at 2026-06-19 14:48:23 on dist-test-slave-2sb1"
server_key: "fcc4428896b7e83c4c65062f5ba39f4e"
server_key_iv: "acb0043d3a195e4785d2f30bafe4d5df"
server_key_version: "kuduclusterkey@0"
I20260619 14:48:23.342902 11590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:48:23.359812 11590 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:48:23.364250 11590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:48:23.364477 11590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:48:23.374974 11590 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35321
I20260619 14:48:23.375049 11657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35321 every 8 connection(s)
I20260619 14:48:23.376339 11590 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:48:23.379590 11658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:48:23.381652 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 11590
I20260619 14:48:23.381791 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/minicluster-data/master-0/wal/instance
I20260619 14:48:23.382088 392 external_mini_cluster.cc:1428] Setting key d6ee68a2bc9dc216664f2c057189b564
I20260619 14:48:23.387953 11658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da0bdfb0db164fda9731b9c2086cbd27: Bootstrap starting.
I20260619 14:48:23.390812 11658 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P da0bdfb0db164fda9731b9c2086cbd27: Neither blocks nor log segments found. Creating new log.
I20260619 14:48:23.391665 11658 log.cc:824] T 00000000000000000000000000000000 P da0bdfb0db164fda9731b9c2086cbd27: Log is configured to *not* fsync() on all Append() calls
Jun 19 14:48:23 dist-test-slave-2sb1 krb5kdc[11146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880503, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:48:23.394589 11658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da0bdfb0db164fda9731b9c2086cbd27: No bootstrap required, opened a new log
I20260619 14:48:23.399624 11658 raft_consensus.cc:348] T 00000000000000000000000000000000 P da0bdfb0db164fda9731b9c2086cbd27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da0bdfb0db164fda9731b9c2086cbd27" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35321 } }
I20260619 14:48:23.400002 11658 raft_consensus.cc:374] T 00000000000000000000000000000000 P da0bdfb0db164fda9731b9c2086cbd27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:48:23.400099 11658 raft_consensus.cc:729] T 00000000000000000000000000000000 P da0bdfb0db164fda9731b9c2086cbd27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da0bdfb0db164fda9731b9c2086cbd27, State: Initialized, Role: FOLLOWER
I20260619 14:48:23.400730 11658 consensus_queue.cc:260] T 00000000000000000000000000000000 P da0bdfb0db164fda9731b9c2086cbd27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da0bdfb0db164fda9731b9c2086cbd27" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35321 } }
I20260619 14:48:23.400944 11658 raft_consensus.cc:388] T 00000000000000000000000000000000 P da0bdfb0db164fda9731b9c2086cbd27 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:48:23.401050 11658 raft_consensus.cc:482] T 00000000000000000000000000000000 P da0bdfb0db164fda9731b9c2086cbd27 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:48:23.401183 11658 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da0bdfb0db164fda9731b9c2086cbd27 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:48:23.402125 11661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:48:23.383713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42066 (local address 127.0.98.62:35321)
0619 14:48:23.384241 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0619 14:48:23.384272 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0619 14:48:23.384323 (+ 51us) server_negotiation.cc:407] Connection header received
0619 14:48:23.385877 (+ 1554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:48:23.385908 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:48:23.386429 (+ 521us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:48:23.386876 (+ 447us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:48:23.387812 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:48:23.389476 (+ 1664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:48:23.390745 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:48:23.391553 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:48:23.391732 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:48:23.395414 (+ 3682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:48:23.395422 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:48:23.395441 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:48:23.395477 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:48:23.398527 (+ 3050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:48:23.399175 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:48:23.399183 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:48:23.399189 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:48:23.399336 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:48:23.399700 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:48:23.399704 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:48:23.399705 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:48:23.400275 (+ 570us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:48:23.400536 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0619 14:48:23.400832 (+ 296us) server_negotiation.cc:299] Negotiation successful
0619 14:48:23.401140 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":152,"threads_started":1}
I20260619 14:48:23.402474 11658 raft_consensus.cc:504] T 00000000000000000000000000000000 P da0bdfb0db164fda9731b9c2086cbd27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da0bdfb0db164fda9731b9c2086cbd27" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35321 } }
I20260619 14:48:23.402853 11658 leader_election.cc:302] T 00000000000000000000000000000000 P da0bdfb0db164fda9731b9c2086cbd27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: da0bdfb0db164fda9731b9c2086cbd27; no voters:
I20260619 14:48:23.403190 11658 leader_election.cc:288] T 00000000000000000000000000000000 P da0bdfb0db164fda9731b9c2086cbd27 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:48:23.403465 11663 raft_consensus.cc:2781] T 00000000000000000000000000000000 P da0bdfb0db164fda9731b9c2086cbd27 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:48:23.403744 11663 raft_consensus.cc:686] T 00000000000000000000000000000000 P da0bdfb0db164fda9731b9c2086cbd27 [term 1 LEADER]: Becoming Leader. State: Replica: da0bdfb0db164fda9731b9c2086cbd27, State: Running, Role: LEADER
I20260619 14:48:23.404137 11663 consensus_queue.cc:237] T 00000000000000000000000000000000 P da0bdfb0db164fda9731b9c2086cbd27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da0bdfb0db164fda9731b9c2086cbd27" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35321 } }
I20260619 14:48:23.404553 11658 sys_catalog.cc:564] T 00000000000000000000000000000000 P da0bdfb0db164fda9731b9c2086cbd27 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:48:23.406239 11664 sys_catalog.cc:455] T 00000000000000000000000000000000 P da0bdfb0db164fda9731b9c2086cbd27 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "da0bdfb0db164fda9731b9c2086cbd27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da0bdfb0db164fda9731b9c2086cbd27" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35321 } } }
I20260619 14:48:23.406388 11664 sys_catalog.cc:458] T 00000000000000000000000000000000 P da0bdfb0db164fda9731b9c2086cbd27 [sys.catalog]: This master's current role is: LEADER
I20260619 14:48:23.406605 11664 sys_catalog.cc:455] T 00000000000000000000000000000000 P da0bdfb0db164fda9731b9c2086cbd27 [sys.catalog]: SysCatalogTable state changed. Reason: New leader da0bdfb0db164fda9731b9c2086cbd27. Latest consensus state: current_term: 1 leader_uuid: "da0bdfb0db164fda9731b9c2086cbd27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da0bdfb0db164fda9731b9c2086cbd27" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35321 } } }
I20260619 14:48:23.406673 11664 sys_catalog.cc:458] T 00000000000000000000000000000000 P da0bdfb0db164fda9731b9c2086cbd27 [sys.catalog]: This master's current role is: LEADER
I20260619 14:48:23.406744 11676 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:48:23.407086 11658 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 14:48:23.410971 11676 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:48:23.428782 11676 catalog_manager.cc:1347] Generated new cluster ID: f3089d86d6954b2ab6a216549f4cf0f4
I20260619 14:48:23.428872 11676 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:48:23.446506 11676 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:48:23.448108 11676 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:48:23.466084 11676 catalog_manager.cc:5868] T 00000000000000000000000000000000 P da0bdfb0db164fda9731b9c2086cbd27: Generated new TSK 0
I20260619 14:48:23.467725 11676 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 14:48:24.207352 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:51683/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35321
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:48:24.347355 11703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:48:24.347681 11703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:48:24.347728 11703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:48:24.353276 11703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:48:24.353359 11703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:48:24.353535 11703 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 14:48:24.360499 11703 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:51683/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35321
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11703
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:48:24.361735 11703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:48:24.363039 11703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:48:24.375002 11711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:48:24.374924 11709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:48:24.375653 11703 server_base.cc:1034] running on GCE node
W20260619 14:48:24.375912 11708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:48:24.376463 11703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:48:24.377339 11703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:48:24.378593 11703 hybrid_clock.cc:648] HybridClock initialized: now 1781880504378552 us; error 66 us; skew 500 ppm
Jun 19 14:48:24 dist-test-slave-2sb1 krb5kdc[11146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880504, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:48:24.382438 11703 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 14:48:24.383682 11703 webserver.cc:460] Webserver started at http://127.0.98.1:43953/ using document root <none> and password file <none>
I20260619 14:48:24.384590 11703 fs_manager.cc:359] Metadata directory not provided
I20260619 14:48:24.384809 11703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:48:24.385156 11703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 14:48:24 dist-test-slave-2sb1 krb5kdc[11146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880504, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 14:48:24.403167 11703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/minicluster-data/ts-0/data/instance:
uuid: "b9e2526d02304506ba61d5770efdfcaa"
format_stamp: "Formatted at 2026-06-19 14:48:24 on dist-test-slave-2sb1"
server_key: "893fbf994ed20d2ec88a5ac4df781598"
server_key_iv: "df653dfbfb511fe67534ab6d2e61ebcb"
server_key_version: "kuduclusterkey@0"
I20260619 14:48:24.404136 11703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b9e2526d02304506ba61d5770efdfcaa"
format_stamp: "Formatted at 2026-06-19 14:48:24 on dist-test-slave-2sb1"
server_key: "893fbf994ed20d2ec88a5ac4df781598"
server_key_iv: "df653dfbfb511fe67534ab6d2e61ebcb"
server_key_version: "kuduclusterkey@0"
I20260619 14:48:24.409132 11703 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 14:48:24.431394 11720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:48:24.433043 11703 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20260619 14:48:24.433205 11703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/minicluster-data/ts-0/wal
uuid: "b9e2526d02304506ba61d5770efdfcaa"
format_stamp: "Formatted at 2026-06-19 14:48:24 on dist-test-slave-2sb1"
server_key: "893fbf994ed20d2ec88a5ac4df781598"
server_key_iv: "df653dfbfb511fe67534ab6d2e61ebcb"
server_key_version: "kuduclusterkey@0"
I20260619 14:48:24.433377 11703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:48:24.459108 11703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:48:24.462625 11703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:48:24.462848 11703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:48:24.463584 11703 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:48:24.464550 11703 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:48:24.464628 11703 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:48:24.464718 11703 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:48:24.464746 11703 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:48:24.478654 11703 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46149
I20260619 14:48:24.479956 11703 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/minicluster-data/ts-0/data/info.pb
I20260619 14:48:24.480887 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 11703
I20260619 14:48:24.481046 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/minicluster-data/ts-0/wal/instance
I20260619 14:48:24.481398 392 external_mini_cluster.cc:1428] Setting key a31595b364f82704e2a070eef5523fb2
Jun 19 14:48:24 dist-test-slave-2sb1 krb5kdc[11146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880504, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:48:24.482973 11833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46149 every 8 connection(s)
I20260619 14:48:24.507249 11837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:48:24.486665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51401 (local address 127.0.98.62:35321)
0619 14:48:24.489052 (+ 2387us) server_negotiation.cc:206] Beginning negotiation
0619 14:48:24.489056 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:48:24.489071 (+ 15us) server_negotiation.cc:407] Connection header received
0619 14:48:24.489113 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:48:24.489116 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:48:24.489166 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:48:24.489255 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:48:24.490996 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:48:24.491891 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:48:24.493971 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:48:24.494343 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:48:24.494436 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:48:24.500906 (+ 6470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:48:24.500911 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:48:24.500914 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:48:24.500951 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:48:24.502949 (+ 1998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:48:24.504078 (+ 1129us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:48:24.504083 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:48:24.504085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:48:24.504230 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:48:24.504986 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:48:24.504990 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:48:24.504992 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:48:24.505359 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:48:24.505478 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 14:48:24.506844 (+ 1366us) server_negotiation.cc:299] Negotiation successful
0619 14:48:24.507018 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2290,"thread_start_us":115,"threads_started":1}
I20260619 14:48:24.509066 11836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:48:24.485513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35321 (local address 127.0.98.1:51401)
0619 14:48:24.487204 (+ 1691us) negotiation.cc:107] Waiting for socket to connect
0619 14:48:24.487285 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0619 14:48:24.488483 (+ 1198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:48:24.489647 (+ 1164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:48:24.489658 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:48:24.490052 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:48:24.490787 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:48:24.490809 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:48:24.492070 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:48:24.492074 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:48:24.493606 (+ 1532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:48:24.493619 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:48:24.494682 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:48:24.494686 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:48:24.494927 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:48:24.498638 (+ 3711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:48:24.498668 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:48:24.500528 (+ 1860us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:48:24.503129 (+ 2601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:48:24.503137 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:48:24.503160 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:48:24.503773 (+ 613us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:48:24.504351 (+ 578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:48:24.504355 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:48:24.504357 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:48:24.504841 (+ 484us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:48:24.505671 (+ 830us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:48:24.505682 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:48:24.506163 (+ 481us) client_negotiation.cc:769] Sending connection context
0619 14:48:24.506425 (+ 262us) client_negotiation.cc:240] Negotiation successful
0619 14:48:24.507802 (+ 1377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1381,"thread_start_us":199,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 14:48:24.510649 11834 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35321
I20260619 14:48:24.511112 11834 heartbeater.cc:464] Registering TS with master...
I20260619 14:48:24.511662 11834 heartbeater.cc:511] Master 127.0.98.62:35321 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 14:48:24.513707 11622 ts_manager.cc:194] Registered new tserver with Master: b9e2526d02304506ba61d5770efdfcaa (127.0.98.1:46149)
I20260619 14:48:24.516466 11622 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:51401
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 14:48:24.558506 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:51683/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35321
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 14:48:24 dist-test-slave-2sb1 krb5kdc[11146](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781880504, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260619 14:48:24.742137 11842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:48:24.742440 11842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:48:24.742486 11842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:48:24.747970 11842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:48:24.748054 11842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:48:24.748119 11842 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 14:48:24.756553 11842 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:51683/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35321
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11842
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:48:24.758083 11842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:48:24.759577 11842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:48:24.777943 11851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:48:24.777957 11848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:48:24.779215 11842 server_base.cc:1034] running on GCE node
W20260619 14:48:24.779603 11849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:48:24.780262 11842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:48:24.781248 11842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:48:24.785327 11842 hybrid_clock.cc:648] HybridClock initialized: now 1781880504785220 us; error 107 us; skew 500 ppm
Jun 19 14:48:24 dist-test-slave-2sb1 krb5kdc[11146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880504, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:48:24.791029 11842 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 14:48:24.792485 11842 webserver.cc:460] Webserver started at http://127.0.98.2:33833/ using document root <none> and password file <none>
I20260619 14:48:24.793334 11842 fs_manager.cc:359] Metadata directory not provided
I20260619 14:48:24.793450 11842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:48:24.793732 11842 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 14:48:24 dist-test-slave-2sb1 krb5kdc[11146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880504, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 14:48:24.815948 11842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/minicluster-data/ts-1/data/instance:
uuid: "9c0117c4102847768af3663e2fe5e3b3"
format_stamp: "Formatted at 2026-06-19 14:48:24 on dist-test-slave-2sb1"
server_key: "4b6acdd88b5ffb32b917bee25ae6479d"
server_key_iv: "70bcdf8e1e3e2122575b63c48c22952c"
server_key_version: "kuduclusterkey@0"
I20260619 14:48:24.818147 11842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/minicluster-data/ts-1/wal/instance:
uuid: "9c0117c4102847768af3663e2fe5e3b3"
format_stamp: "Formatted at 2026-06-19 14:48:24 on dist-test-slave-2sb1"
server_key: "4b6acdd88b5ffb32b917bee25ae6479d"
server_key_iv: "70bcdf8e1e3e2122575b63c48c22952c"
server_key_version: "kuduclusterkey@0"
I20260619 14:48:24.824396 11842 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260619 14:48:24.846666 11858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:48:24.848610 11842 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 14:48:24.848834 11842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/minicluster-data/ts-1/wal
uuid: "9c0117c4102847768af3663e2fe5e3b3"
format_stamp: "Formatted at 2026-06-19 14:48:24 on dist-test-slave-2sb1"
server_key: "4b6acdd88b5ffb32b917bee25ae6479d"
server_key_iv: "70bcdf8e1e3e2122575b63c48c22952c"
server_key_version: "kuduclusterkey@0"
I20260619 14:48:24.849011 11842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:48:24.873160 11842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:48:24.878012 11842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:48:24.878399 11842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:48:24.879550 11842 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:48:24.880975 11842 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:48:24.881059 11842 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:48:24.881140 11842 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:48:24.881178 11842 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:48:24.899477 11842 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40985
I20260619 14:48:24.901041 11842 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/minicluster-data/ts-1/data/info.pb
I20260619 14:48:24.901774 11971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40985 every 8 connection(s)
I20260619 14:48:24.910349 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 11842
I20260619 14:48:24.910517 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/minicluster-data/ts-1/wal/instance
I20260619 14:48:24.910899 392 external_mini_cluster.cc:1428] Setting key 6140e7f2a175d118933d94c870cc6db7
Jun 19 14:48:24 dist-test-slave-2sb1 krb5kdc[11146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880504, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:48:24.926944 11837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:48:24.905963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52811 (local address 127.0.98.62:35321)
0619 14:48:24.906118 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 14:48:24.906125 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:48:24.907671 (+ 1546us) server_negotiation.cc:407] Connection header received
0619 14:48:24.909251 (+ 1580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:48:24.909256 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:48:24.909331 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:48:24.909403 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:48:24.911251 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:48:24.911966 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:48:24.916693 (+ 4727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:48:24.917132 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:48:24.917227 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:48:24.920932 (+ 3705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:48:24.920937 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:48:24.920940 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:48:24.920983 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:48:24.923551 (+ 2568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:48:24.924497 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:48:24.924503 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:48:24.924506 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:48:24.924617 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:48:24.925170 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:48:24.925174 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:48:24.925175 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:48:24.925470 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:48:24.925564 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 14:48:24.926494 (+ 930us) server_negotiation.cc:299] Negotiation successful
0619 14:48:24.926731 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 14:48:24.932144 11974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:48:24.904580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35321 (local address 127.0.98.2:52811)
0619 14:48:24.907450 (+ 2870us) negotiation.cc:107] Waiting for socket to connect
0619 14:48:24.907528 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0619 14:48:24.908904 (+ 1376us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:48:24.909934 (+ 1030us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:48:24.909945 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:48:24.910351 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:48:24.911067 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:48:24.911086 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:48:24.912122 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:48:24.912125 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:48:24.916425 (+ 4300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:48:24.916449 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:48:24.917515 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:48:24.917520 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:48:24.917813 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:48:24.918637 (+ 824us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:48:24.918663 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:48:24.920606 (+ 1943us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:48:24.923791 (+ 3185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:48:24.923799 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:48:24.923823 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:48:24.924329 (+ 506us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:48:24.924798 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:48:24.924802 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:48:24.924804 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:48:24.925032 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:48:24.925643 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:48:24.925651 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:48:24.926124 (+ 473us) client_negotiation.cc:769] Sending connection context
0619 14:48:24.927255 (+ 1131us) client_negotiation.cc:240] Negotiation successful
0619 14:48:24.927710 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2521,"thread_start_us":186,"threads_started":1}
I20260619 14:48:24.933982 11972 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35321
I20260619 14:48:24.934309 11972 heartbeater.cc:464] Registering TS with master...
I20260619 14:48:24.935031 11972 heartbeater.cc:511] Master 127.0.98.62:35321 requested a full tablet report, sending...
I20260619 14:48:24.936527 11622 ts_manager.cc:194] Registered new tserver with Master: 9c0117c4102847768af3663e2fe5e3b3 (127.0.98.2:40985)
I20260619 14:48:24.937464 11622 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52811
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 14:48:24.994032 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:51683/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35321
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:48:25.211309 11977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:48:25.211932 11977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:48:25.212097 11977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:48:25.219631 11977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:48:25.219745 11977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:48:25.219853 11977 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 14:48:25.228845 11977 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:51683/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35321
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11977
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:48:25.230618 11977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:48:25.232249 11977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:48:25.248973 11985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:48:25.249712 11983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:48:25.251044 11977 server_base.cc:1034] running on GCE node
W20260619 14:48:25.251698 11982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:48:25.252305 11977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:48:25.253243 11977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
Jun 19 14:48:25 dist-test-slave-2sb1 krb5kdc[11146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880504, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 14:48:25.256592 11977 hybrid_clock.cc:648] HybridClock initialized: now 1781880505256470 us; error 122 us; skew 500 ppm
Jun 19 14:48:25 dist-test-slave-2sb1 krb5kdc[11146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880505, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:48:25.261600 11977 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 14:48:25.263113 11977 webserver.cc:460] Webserver started at http://127.0.98.3:35383/ using document root <none> and password file <none>
I20260619 14:48:25.264132 11977 fs_manager.cc:359] Metadata directory not provided
I20260619 14:48:25.264323 11977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:48:25.264688 11977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 14:48:25 dist-test-slave-2sb1 krb5kdc[11146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880505, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 14:48:25.299082 11977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/minicluster-data/ts-2/data/instance:
uuid: "736cc5237b2d4787a868ba6f7e582985"
format_stamp: "Formatted at 2026-06-19 14:48:25 on dist-test-slave-2sb1"
server_key: "1770fbe4484dc63f54298711293a468c"
server_key_iv: "21a692bc9d287d9b8c4b17b9b54ca720"
server_key_version: "kuduclusterkey@0"
I20260619 14:48:25.301592 11977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/minicluster-data/ts-2/wal/instance:
uuid: "736cc5237b2d4787a868ba6f7e582985"
format_stamp: "Formatted at 2026-06-19 14:48:25 on dist-test-slave-2sb1"
server_key: "1770fbe4484dc63f54298711293a468c"
server_key_iv: "21a692bc9d287d9b8c4b17b9b54ca720"
server_key_version: "kuduclusterkey@0"
I20260619 14:48:25.328461 11977 fs_manager.cc:691] Time spent creating directory manager: real 0.026s user 0.005s sys 0.000s
I20260619 14:48:25.361151 11992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:48:25.374747 11977 fs_manager.cc:725] Time spent opening block manager: real 0.027s user 0.003s sys 0.000s
I20260619 14:48:25.374950 11977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/minicluster-data/ts-2/wal
uuid: "736cc5237b2d4787a868ba6f7e582985"
format_stamp: "Formatted at 2026-06-19 14:48:25 on dist-test-slave-2sb1"
server_key: "1770fbe4484dc63f54298711293a468c"
server_key_iv: "21a692bc9d287d9b8c4b17b9b54ca720"
server_key_version: "kuduclusterkey@0"
I20260619 14:48:25.375129 11977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:48:25.412225 11977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:48:25.416970 11977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:48:25.417353 11977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:48:25.418267 11977 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:48:25.419574 11977 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:48:25.419643 11977 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:48:25.419697 11977 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:48:25.419735 11977 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:48:25.431772 11977 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37843
I20260619 14:48:25.431820 12105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37843 every 8 connection(s)
I20260619 14:48:25.433079 11977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/minicluster-data/ts-2/data/info.pb
I20260619 14:48:25.442401 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 11977
I20260619 14:48:25.442548 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781880383132139-392-0/minicluster-data/ts-2/wal/instance
I20260619 14:48:25.442874 392 external_mini_cluster.cc:1428] Setting key 3d5ad1ce6267ec157e03ad3b03106ca6
Jun 19 14:48:25 dist-test-slave-2sb1 krb5kdc[11146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880505, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:48:25.453191 12108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:48:25.435232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60039 (local address 127.0.98.62:35321)
0619 14:48:25.435502 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0619 14:48:25.435506 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:48:25.436526 (+ 1020us) server_negotiation.cc:407] Connection header received
0619 14:48:25.437935 (+ 1409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:48:25.437940 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:48:25.438002 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:48:25.438064 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:48:25.439745 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:48:25.440575 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:48:25.443702 (+ 3127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:48:25.444148 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:48:25.444267 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:48:25.447782 (+ 3515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:48:25.447787 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:48:25.447789 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:48:25.447821 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:48:25.449953 (+ 2132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:48:25.450728 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:48:25.450733 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:48:25.450735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:48:25.450839 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:48:25.451303 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:48:25.451307 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:48:25.451309 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:48:25.451968 (+ 659us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:48:25.452081 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 14:48:25.452822 (+ 741us) server_negotiation.cc:299] Negotiation successful
0619 14:48:25.452995 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":84,"threads_started":1}
I20260619 14:48:25.454630 12109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:48:25.435746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35321 (local address 127.0.98.3:60039)
0619 14:48:25.436337 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0619 14:48:25.436408 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0619 14:48:25.437593 (+ 1185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:48:25.438493 (+ 900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:48:25.438503 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:48:25.438877 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:48:25.439555 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:48:25.439573 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:48:25.440782 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:48:25.440785 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:48:25.443511 (+ 2726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:48:25.443530 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:48:25.444539 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:48:25.444545 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:48:25.444854 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:48:25.445641 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:48:25.445667 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:48:25.447537 (+ 1870us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:48:25.450126 (+ 2589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:48:25.450135 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:48:25.450156 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:48:25.450576 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:48:25.450948 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:48:25.450951 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:48:25.450953 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:48:25.451169 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:48:25.452106 (+ 937us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:48:25.452114 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:48:25.452488 (+ 374us) client_negotiation.cc:769] Sending connection context
0619 14:48:25.452741 (+ 253us) client_negotiation.cc:240] Negotiation successful
0619 14:48:25.453105 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":158,"threads_started":1}
I20260619 14:48:25.456194 12106 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35321
I20260619 14:48:25.456457 12106 heartbeater.cc:464] Registering TS with master...
I20260619 14:48:25.457193 12106 heartbeater.cc:511] Master 127.0.98.62:35321 requested a full tablet report, sending...
I20260619 14:48:25.458455 11622 ts_manager.cc:194] Registered new tserver with Master: 736cc5237b2d4787a868ba6f7e582985 (127.0.98.3:37843)
I20260619 14:48:25.459532 11622 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:60039
I20260619 14:48:25.470067 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:48:25.492035 12108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:48:25.472922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42076 (local address 127.0.98.62:35321)
0619 14:48:25.473056 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 14:48:25.473060 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:48:25.481354 (+ 8294us) server_negotiation.cc:407] Connection header received
0619 14:48:25.481605 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:48:25.481609 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:48:25.481667 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:48:25.481728 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:48:25.482917 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:48:25.483761 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:48:25.485173 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:48:25.485527 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:48:25.485613 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:48:25.487617 (+ 2004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:48:25.487622 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:48:25.487625 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:48:25.487659 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:48:25.489641 (+ 1982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:48:25.490386 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:48:25.490390 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:48:25.490392 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:48:25.490502 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:48:25.490879 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:48:25.490883 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:48:25.490885 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:48:25.491141 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:48:25.491293 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 14:48:25.491685 (+ 392us) server_negotiation.cc:299] Negotiation successful
0619 14:48:25.491862 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 14:48:25.496030 11622 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42076:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:48:25.500813 11622 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:48:25.523397 11834 heartbeater.cc:503] Master 127.0.98.62:35321 was elected leader, sending a full tablet report...
I20260619 14:48:25.526988 12123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:48:25.515313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46149 (local address 127.0.0.1:48564)
0619 14:48:25.516977 (+ 1664us) negotiation.cc:107] Waiting for socket to connect
0619 14:48:25.516994 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 14:48:25.517089 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:48:25.519328 (+ 2239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:48:25.519331 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:48:25.519346 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:48:25.519473 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:48:25.519481 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:48:25.521428 (+ 1947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:48:25.521433 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:48:25.522790 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:48:25.522802 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:48:25.526362 (+ 3560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:48:25.526366 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:48:25.526513 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:48:25.526559 (+ 46us) client_negotiation.cc:769] Sending connection context
0619 14:48:25.526682 (+ 123us) client_negotiation.cc:240] Negotiation successful
0619 14:48:25.526818 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1595,"mutex_wait_us":23,"thread_start_us":122,"threads_started":1}
I20260619 14:48:25.527182 12119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:48:25.513363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37843 (local address 127.0.0.1:41196)
0619 14:48:25.513891 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0619 14:48:25.513958 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0619 14:48:25.514107 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:48:25.516000 (+ 1893us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:48:25.516006 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:48:25.516034 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:48:25.516205 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:48:25.516213 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:48:25.519029 (+ 2816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:48:25.519033 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:48:25.520769 (+ 1736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:48:25.520783 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:48:25.526738 (+ 5955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:48:25.526742 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:48:25.526864 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:48:25.526891 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 14:48:25.526945 (+ 54us) client_negotiation.cc:240] Negotiation successful
0619 14:48:25.527026 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"mutex_wait_us":3,"thread_start_us":176,"threads_started":1}
I20260619 14:48:25.527830 12120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:48:25.514358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41196 (local address 127.0.98.3:37843)
0619 14:48:25.515432 (+ 1074us) server_negotiation.cc:206] Beginning negotiation
0619 14:48:25.515445 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 14:48:25.515470 (+ 25us) server_negotiation.cc:407] Connection header received
0619 14:48:25.515552 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:48:25.515560 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:48:25.515801 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:48:25.516394 (+ 593us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:48:25.516542 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:48:25.518869 (+ 2327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:48:25.522478 (+ 3609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:48:25.526577 (+ 4099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:48:25.527317 (+ 740us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:48:25.527484 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0619 14:48:25.527551 (+ 67us) server_negotiation.cc:299] Negotiation successful
0619 14:48:25.527664 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":945,"thread_start_us":158,"threads_started":1}
I20260619 14:48:25.528905 12122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:48:25.515506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48564 (local address 127.0.98.1:46149)
0619 14:48:25.518626 (+ 3120us) server_negotiation.cc:206] Beginning negotiation
0619 14:48:25.518637 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 14:48:25.518661 (+ 24us) server_negotiation.cc:407] Connection header received
0619 14:48:25.518757 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:48:25.518765 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:48:25.518970 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:48:25.519657 (+ 687us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:48:25.519828 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:48:25.521273 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:48:25.524923 (+ 3650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:48:25.525886 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:48:25.526008 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:48:25.526178 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0619 14:48:25.528601 (+ 2423us) server_negotiation.cc:299] Negotiation successful
0619 14:48:25.528750 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3003,"thread_start_us":99,"threads_started":1}
I20260619 14:48:25.531601 12022 tablet_service.cc:1467] Processing CreateTablet for tablet a6347e081ff14b9789c4f476ad9816ec (DEFAULT_TABLE table=test-table [id=e57d0e58fa324c5d81b23fa7b7537353]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:48:25.532992 12121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:48:25.516042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40985 (local address 127.0.0.1:42526)
0619 14:48:25.517500 (+ 1458us) negotiation.cc:107] Waiting for socket to connect
0619 14:48:25.517517 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 14:48:25.517629 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:48:25.519119 (+ 1490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:48:25.519122 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:48:25.519143 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:48:25.522931 (+ 3788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:48:25.522937 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:48:25.524438 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:48:25.524442 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:48:25.528422 (+ 3980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:48:25.528434 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:48:25.531795 (+ 3361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:48:25.531800 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:48:25.531936 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:48:25.531965 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 14:48:25.532730 (+ 765us) client_negotiation.cc:240] Negotiation successful
0619 14:48:25.532841 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1391,"mutex_wait_us":3009,"thread_start_us":91,"threads_started":1}
I20260619 14:48:25.533028 12022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6347e081ff14b9789c4f476ad9816ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:48:25.533366 12124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:48:25.516523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42526 (local address 127.0.98.2:40985)
0619 14:48:25.517346 (+ 823us) server_negotiation.cc:206] Beginning negotiation
0619 14:48:25.517355 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 14:48:25.517767 (+ 412us) server_negotiation.cc:407] Connection header received
0619 14:48:25.517858 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:48:25.517865 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:48:25.518114 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:48:25.518292 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:48:25.523065 (+ 4773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:48:25.524265 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:48:25.529186 (+ 4921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:48:25.531393 (+ 2207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:48:25.531500 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:48:25.531657 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 14:48:25.532114 (+ 457us) server_negotiation.cc:299] Negotiation successful
0619 14:48:25.532234 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":700,"thread_start_us":92,"threads_started":1}
I20260619 14:48:25.534861 11759 tablet_service.cc:1467] Processing CreateTablet for tablet a6347e081ff14b9789c4f476ad9816ec (DEFAULT_TABLE table=test-table [id=e57d0e58fa324c5d81b23fa7b7537353]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:48:25.536237 11905 tablet_service.cc:1467] Processing CreateTablet for tablet a6347e081ff14b9789c4f476ad9816ec (DEFAULT_TABLE table=test-table [id=e57d0e58fa324c5d81b23fa7b7537353]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:48:25.537468 11759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6347e081ff14b9789c4f476ad9816ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:48:25.537446 11905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6347e081ff14b9789c4f476ad9816ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:48:25.547385 12126 tablet_bootstrap.cc:492] T a6347e081ff14b9789c4f476ad9816ec P b9e2526d02304506ba61d5770efdfcaa: Bootstrap starting.
I20260619 14:48:25.548455 12125 tablet_bootstrap.cc:492] T a6347e081ff14b9789c4f476ad9816ec P 9c0117c4102847768af3663e2fe5e3b3: Bootstrap starting.
I20260619 14:48:25.550508 12126 tablet_bootstrap.cc:654] T a6347e081ff14b9789c4f476ad9816ec P b9e2526d02304506ba61d5770efdfcaa: Neither blocks nor log segments found. Creating new log.
I20260619 14:48:25.551033 12125 tablet_bootstrap.cc:654] T a6347e081ff14b9789c4f476ad9816ec P 9c0117c4102847768af3663e2fe5e3b3: Neither blocks nor log segments found. Creating new log.
I20260619 14:48:25.551048 12127 tablet_bootstrap.cc:492] T a6347e081ff14b9789c4f476ad9816ec P 736cc5237b2d4787a868ba6f7e582985: Bootstrap starting.
I20260619 14:48:25.552095 12125 log.cc:824] T a6347e081ff14b9789c4f476ad9816ec P 9c0117c4102847768af3663e2fe5e3b3: Log is configured to *not* fsync() on all Append() calls
I20260619 14:48:25.554266 12127 tablet_bootstrap.cc:654] T a6347e081ff14b9789c4f476ad9816ec P 736cc5237b2d4787a868ba6f7e582985: Neither blocks nor log segments found. Creating new log.
I20260619 14:48:25.555294 12126 log.cc:824] T a6347e081ff14b9789c4f476ad9816ec P b9e2526d02304506ba61d5770efdfcaa: Log is configured to *not* fsync() on all Append() calls
I20260619 14:48:25.556241 12127 log.cc:824] T a6347e081ff14b9789c4f476ad9816ec P 736cc5237b2d4787a868ba6f7e582985: Log is configured to *not* fsync() on all Append() calls
I20260619 14:48:25.559469 12127 tablet_bootstrap.cc:492] T a6347e081ff14b9789c4f476ad9816ec P 736cc5237b2d4787a868ba6f7e582985: No bootstrap required, opened a new log
I20260619 14:48:25.559782 12127 ts_tablet_manager.cc:1397] T a6347e081ff14b9789c4f476ad9816ec P 736cc5237b2d4787a868ba6f7e582985: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260619 14:48:25.561529 12126 tablet_bootstrap.cc:492] T a6347e081ff14b9789c4f476ad9816ec P b9e2526d02304506ba61d5770efdfcaa: No bootstrap required, opened a new log
I20260619 14:48:25.561863 12126 ts_tablet_manager.cc:1397] T a6347e081ff14b9789c4f476ad9816ec P b9e2526d02304506ba61d5770efdfcaa: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.002s
I20260619 14:48:25.565414 12127 raft_consensus.cc:348] T a6347e081ff14b9789c4f476ad9816ec P 736cc5237b2d4787a868ba6f7e582985 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "736cc5237b2d4787a868ba6f7e582985" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37843 } } peers { permanent_uuid: "b9e2526d02304506ba61d5770efdfcaa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46149 } } peers { permanent_uuid: "9c0117c4102847768af3663e2fe5e3b3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40985 } }
I20260619 14:48:25.565754 12127 raft_consensus.cc:374] T a6347e081ff14b9789c4f476ad9816ec P 736cc5237b2d4787a868ba6f7e582985 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:48:25.565814 12127 raft_consensus.cc:729] T a6347e081ff14b9789c4f476ad9816ec P 736cc5237b2d4787a868ba6f7e582985 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 736cc5237b2d4787a868ba6f7e582985, State: Initialized, Role: FOLLOWER
I20260619 14:48:25.567248 12126 raft_consensus.cc:348] T a6347e081ff14b9789c4f476ad9816ec P b9e2526d02304506ba61d5770efdfcaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "736cc5237b2d4787a868ba6f7e582985" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37843 } } peers { permanent_uuid: "b9e2526d02304506ba61d5770efdfcaa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46149 } } peers { permanent_uuid: "9c0117c4102847768af3663e2fe5e3b3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40985 } }
I20260619 14:48:25.567588 12127 consensus_queue.cc:260] T a6347e081ff14b9789c4f476ad9816ec P 736cc5237b2d4787a868ba6f7e582985 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "736cc5237b2d4787a868ba6f7e582985" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37843 } } peers { permanent_uuid: "b9e2526d02304506ba61d5770efdfcaa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46149 } } peers { permanent_uuid: "9c0117c4102847768af3663e2fe5e3b3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40985 } }
I20260619 14:48:25.568027 12126 raft_consensus.cc:374] T a6347e081ff14b9789c4f476ad9816ec P b9e2526d02304506ba61d5770efdfcaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:48:25.568147 12126 raft_consensus.cc:729] T a6347e081ff14b9789c4f476ad9816ec P b9e2526d02304506ba61d5770efdfcaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9e2526d02304506ba61d5770efdfcaa, State: Initialized, Role: FOLLOWER
I20260619 14:48:25.568843 12126 consensus_queue.cc:260] T a6347e081ff14b9789c4f476ad9816ec P b9e2526d02304506ba61d5770efdfcaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "736cc5237b2d4787a868ba6f7e582985" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37843 } } peers { permanent_uuid: "b9e2526d02304506ba61d5770efdfcaa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46149 } } peers { permanent_uuid: "9c0117c4102847768af3663e2fe5e3b3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40985 } }
I20260619 14:48:25.569612 12127 ts_tablet_manager.cc:1428] T a6347e081ff14b9789c4f476ad9816ec P 736cc5237b2d4787a868ba6f7e582985: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20260619 14:48:25.569864 12126 ts_tablet_manager.cc:1428] T a6347e081ff14b9789c4f476ad9816ec P b9e2526d02304506ba61d5770efdfcaa: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260619 14:48:25.571516 12125 tablet_bootstrap.cc:492] T a6347e081ff14b9789c4f476ad9816ec P 9c0117c4102847768af3663e2fe5e3b3: No bootstrap required, opened a new log
I20260619 14:48:25.571873 12125 ts_tablet_manager.cc:1397] T a6347e081ff14b9789c4f476ad9816ec P 9c0117c4102847768af3663e2fe5e3b3: Time spent bootstrapping tablet: real 0.026s user 0.006s sys 0.000s
I20260619 14:48:25.575742 12106 heartbeater.cc:503] Master 127.0.98.62:35321 was elected leader, sending a full tablet report...
I20260619 14:48:25.575834 12131 raft_consensus.cc:482] T a6347e081ff14b9789c4f476ad9816ec P 736cc5237b2d4787a868ba6f7e582985 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:48:25.576009 12131 raft_consensus.cc:504] T a6347e081ff14b9789c4f476ad9816ec P 736cc5237b2d4787a868ba6f7e582985 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "736cc5237b2d4787a868ba6f7e582985" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37843 } } peers { permanent_uuid: "b9e2526d02304506ba61d5770efdfcaa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46149 } } peers { permanent_uuid: "9c0117c4102847768af3663e2fe5e3b3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40985 } }
I20260619 14:48:25.576771 12125 raft_consensus.cc:348] T a6347e081ff14b9789c4f476ad9816ec P 9c0117c4102847768af3663e2fe5e3b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "736cc5237b2d4787a868ba6f7e582985" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37843 } } peers { permanent_uuid: "b9e2526d02304506ba61d5770efdfcaa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46149 } } peers { permanent_uuid: "9c0117c4102847768af3663e2fe5e3b3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40985 } }
I20260619 14:48:25.577134 12125 raft_consensus.cc:374] T a6347e081ff14b9789c4f476ad9816ec P 9c0117c4102847768af3663e2fe5e3b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:48:25.577232 12125 raft_consensus.cc:729] T a6347e081ff14b9789c4f476ad9816ec P 9c0117c4102847768af3663e2fe5e3b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c0117c4102847768af3663e2fe5e3b3, State: Initialized, Role: FOLLOWER
I20260619 14:48:25.577407 12131 leader_election.cc:288] T a6347e081ff14b9789c4f476ad9816ec P 736cc5237b2d4787a868ba6f7e582985 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9e2526d02304506ba61d5770efdfcaa (127.0.98.1:46149), 9c0117c4102847768af3663e2fe5e3b3 (127.0.98.2:40985)
I20260619 14:48:25.579232 12125 consensus_queue.cc:260] T a6347e081ff14b9789c4f476ad9816ec P 9c0117c4102847768af3663e2fe5e3b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "736cc5237b2d4787a868ba6f7e582985" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37843 } } peers { permanent_uuid: "b9e2526d02304506ba61d5770efdfcaa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46149 } } peers { permanent_uuid: "9c0117c4102847768af3663e2fe5e3b3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40985 } }
I20260619 14:48:25.580629 12125 ts_tablet_manager.cc:1428] T a6347e081ff14b9789c4f476ad9816ec P 9c0117c4102847768af3663e2fe5e3b3: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260619 14:48:25.582626 11972 heartbeater.cc:503] Master 127.0.98.62:35321 was elected leader, sending a full tablet report...
I20260619 14:48:25.587059 12124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:48:25.578449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34337 (local address 127.0.98.2:40985)
0619 14:48:25.578588 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0619 14:48:25.578592 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:48:25.581522 (+ 2930us) server_negotiation.cc:407] Connection header received
0619 14:48:25.581726 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:48:25.581729 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:48:25.581774 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:48:25.581818 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:48:25.582183 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:48:25.582944 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:48:25.585710 (+ 2766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:48:25.586286 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:48:25.586369 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:48:25.586439 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0619 14:48:25.586825 (+ 386us) server_negotiation.cc:299] Negotiation successful
0619 14:48:25.586921 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 14:48:25.587738 12134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:48:25.577888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40985 (local address 127.0.98.3:34337)
0619 14:48:25.581322 (+ 3434us) negotiation.cc:107] Waiting for socket to connect
0619 14:48:25.581339 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 14:48:25.581608 (+ 269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:48:25.581935 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:48:25.581938 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:48:25.581951 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:48:25.582063 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:48:25.582069 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:48:25.584326 (+ 2257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:48:25.584330 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:48:25.585557 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:48:25.585570 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:48:25.586546 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:48:25.586549 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:48:25.586656 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:48:25.586694 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 14:48:25.587485 (+ 791us) client_negotiation.cc:240] Negotiation successful
0619 14:48:25.587572 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3349,"thread_start_us":110,"threads_started":1}
I20260619 14:48:25.588511 11921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6347e081ff14b9789c4f476ad9816ec" candidate_uuid: "736cc5237b2d4787a868ba6f7e582985" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c0117c4102847768af3663e2fe5e3b3" is_pre_election: true
I20260619 14:48:25.588912 11921 raft_consensus.cc:2454] T a6347e081ff14b9789c4f476ad9816ec P 9c0117c4102847768af3663e2fe5e3b3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 736cc5237b2d4787a868ba6f7e582985 in term 0.
I20260619 14:48:25.589290 12109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:48:25.577591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46149 (local address 127.0.98.3:58267)
0619 14:48:25.577800 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0619 14:48:25.577818 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 14:48:25.577900 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:48:25.579508 (+ 1608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:48:25.579512 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:48:25.579531 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:48:25.579664 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:48:25.579676 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:48:25.580934 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:48:25.580937 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:48:25.582318 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:48:25.582326 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:48:25.588828 (+ 6502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:48:25.588832 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:48:25.588953 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:48:25.588981 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 14:48:25.589064 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 14:48:25.589148 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108}
I20260619 14:48:25.589442 11996 leader_election.cc:302] T a6347e081ff14b9789c4f476ad9816ec P 736cc5237b2d4787a868ba6f7e582985 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 736cc5237b2d4787a868ba6f7e582985, 9c0117c4102847768af3663e2fe5e3b3; no voters:
I20260619 14:48:25.589805 12132 raft_consensus.cc:2781] T a6347e081ff14b9789c4f476ad9816ec P 736cc5237b2d4787a868ba6f7e582985 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:48:25.589915 12132 raft_consensus.cc:482] T a6347e081ff14b9789c4f476ad9816ec P 736cc5237b2d4787a868ba6f7e582985 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:48:25.589969 12132 raft_consensus.cc:3037] T a6347e081ff14b9789c4f476ad9816ec P 736cc5237b2d4787a868ba6f7e582985 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:48:25.591403 12122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:48:25.578314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58267 (local address 127.0.98.1:46149)
0619 14:48:25.578437 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 14:48:25.578442 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:48:25.578460 (+ 18us) server_negotiation.cc:407] Connection header received
0619 14:48:25.578509 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:48:25.578512 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:48:25.578559 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:48:25.578652 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:48:25.580043 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:48:25.580787 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:48:25.585079 (+ 4292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:48:25.585730 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:48:25.585829 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:48:25.585911 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0619 14:48:25.591077 (+ 5166us) server_negotiation.cc:299] Negotiation successful
0619 14:48:25.591185 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260619 14:48:25.591472 12132 raft_consensus.cc:504] T a6347e081ff14b9789c4f476ad9816ec P 736cc5237b2d4787a868ba6f7e582985 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "736cc5237b2d4787a868ba6f7e582985" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37843 } } peers { permanent_uuid: "b9e2526d02304506ba61d5770efdfcaa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46149 } } peers { permanent_uuid: "9c0117c4102847768af3663e2fe5e3b3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40985 } }
I20260619 14:48:25.591907 12132 leader_election.cc:288] T a6347e081ff14b9789c4f476ad9816ec P 736cc5237b2d4787a868ba6f7e582985 [CANDIDATE]: Term 1 election: Requested vote from peers b9e2526d02304506ba61d5770efdfcaa (127.0.98.1:46149), 9c0117c4102847768af3663e2fe5e3b3 (127.0.98.2:40985)
I20260619 14:48:25.592229 11778 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6347e081ff14b9789c4f476ad9816ec" candidate_uuid: "736cc5237b2d4787a868ba6f7e582985" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9e2526d02304506ba61d5770efdfcaa" is_pre_election: true
I20260619 14:48:25.592350 11788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6347e081ff14b9789c4f476ad9816ec" candidate_uuid: "736cc5237b2d4787a868ba6f7e582985" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9e2526d02304506ba61d5770efdfcaa"
I20260619 14:48:25.592581 11788 raft_consensus.cc:3037] T a6347e081ff14b9789c4f476ad9816ec P b9e2526d02304506ba61d5770efdfcaa [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:48:25.594275 11788 raft_consensus.cc:2454] T a6347e081ff14b9789c4f476ad9816ec P b9e2526d02304506ba61d5770efdfcaa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 736cc5237b2d4787a868ba6f7e582985 in term 1.
I20260619 14:48:25.594655 11993 leader_election.cc:302] T a6347e081ff14b9789c4f476ad9816ec P 736cc5237b2d4787a868ba6f7e582985 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 736cc5237b2d4787a868ba6f7e582985, b9e2526d02304506ba61d5770efdfcaa; no voters:
I20260619 14:48:25.594873 12132 raft_consensus.cc:2781] T a6347e081ff14b9789c4f476ad9816ec P 736cc5237b2d4787a868ba6f7e582985 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:48:25.594969 12132 raft_consensus.cc:686] T a6347e081ff14b9789c4f476ad9816ec P 736cc5237b2d4787a868ba6f7e582985 [term 1 LEADER]: Becoming Leader. State: Replica: 736cc5237b2d4787a868ba6f7e582985, State: Running, Role: LEADER
I20260619 14:48:25.595350 12132 consensus_queue.cc:237] T a6347e081ff14b9789c4f476ad9816ec P 736cc5237b2d4787a868ba6f7e582985 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "736cc5237b2d4787a868ba6f7e582985" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37843 } } peers { permanent_uuid: "b9e2526d02304506ba61d5770efdfcaa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46149 } } peers { permanent_uuid: "9c0117c4102847768af3663e2fe5e3b3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40985 } }
I20260619 14:48:25.595921 11921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6347e081ff14b9789c4f476ad9816ec" candidate_uuid: "736cc5237b2d4787a868ba6f7e582985" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c0117c4102847768af3663e2fe5e3b3"
I20260619 14:48:25.596076 11921 raft_consensus.cc:3037] T a6347e081ff14b9789c4f476ad9816ec P 9c0117c4102847768af3663e2fe5e3b3 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:48:25.598366 11921 raft_consensus.cc:2454] T a6347e081ff14b9789c4f476ad9816ec P 9c0117c4102847768af3663e2fe5e3b3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 736cc5237b2d4787a868ba6f7e582985 in term 1.
I20260619 14:48:25.600963 11622 catalog_manager.cc:5495] T a6347e081ff14b9789c4f476ad9816ec P 736cc5237b2d4787a868ba6f7e582985 reported cstate change: term changed from 0 to 1, leader changed from <none> to 736cc5237b2d4787a868ba6f7e582985 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "736cc5237b2d4787a868ba6f7e582985" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "736cc5237b2d4787a868ba6f7e582985" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9e2526d02304506ba61d5770efdfcaa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c0117c4102847768af3663e2fe5e3b3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40985 } health_report { overall_health: UNKNOWN } } }
I20260619 14:48:25.641813 12120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:48:25.634260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41200 (local address 127.0.98.3:37843)
0619 14:48:25.635056 (+ 796us) server_negotiation.cc:206] Beginning negotiation
0619 14:48:25.635062 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:48:25.635083 (+ 21us) server_negotiation.cc:407] Connection header received
0619 14:48:25.635193 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:48:25.635197 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:48:25.635279 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:48:25.635422 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:48:25.636605 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:48:25.637588 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:48:25.639170 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:48:25.639570 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:48:25.639690 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:48:25.640228 (+ 538us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:48:25.640894 (+ 666us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:48:25.641021 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 14:48:25.641547 (+ 526us) server_negotiation.cc:299] Negotiation successful
0619 14:48:25.641624 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":683}
I20260619 14:48:25.652740 11788 raft_consensus.cc:1261] T a6347e081ff14b9789c4f476ad9816ec P b9e2526d02304506ba61d5770efdfcaa [term 1 FOLLOWER]: Refusing update from remote peer 736cc5237b2d4787a868ba6f7e582985: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:48:25.652834 11921 raft_consensus.cc:1261] T a6347e081ff14b9789c4f476ad9816ec P 9c0117c4102847768af3663e2fe5e3b3 [term 1 FOLLOWER]: Refusing update from remote peer 736cc5237b2d4787a868ba6f7e582985: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:48:25.653559 12132 consensus_queue.cc:1035] T a6347e081ff14b9789c4f476ad9816ec P 736cc5237b2d4787a868ba6f7e582985 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9e2526d02304506ba61d5770efdfcaa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:48:25.654078 12132 consensus_queue.cc:1035] T a6347e081ff14b9789c4f476ad9816ec P 736cc5237b2d4787a868ba6f7e582985 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c0117c4102847768af3663e2fe5e3b3" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 14:48:25.659528 11973 tablet.cc:2369] T a6347e081ff14b9789c4f476ad9816ec P 9c0117c4102847768af3663e2fe5e3b3: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 19 14:48:25 dist-test-slave-2sb1 krb5kdc[11146](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880504, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 14:48:25.679522 12142 mvcc.cc:205] Tried to move back new op lower bound from 7298582551143936000 to 7298582550922244096. Current Snapshot: MvccSnapshot[applied={T|T < 7298582551143936000}]
I20260619 14:48:25.696801 12140 mvcc.cc:205] Tried to move back new op lower bound from 7298582551143936000 to 7298582550922244096. Current Snapshot: MvccSnapshot[applied={T|T < 7298582551143936000}]
I20260619 14:48:25.699524 11622 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42076:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:48:25.699726 11622 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42076:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:48:25.705462 11622 catalog_manager.cc:5782] T 00000000000000000000000000000000 P da0bdfb0db164fda9731b9c2086cbd27: Sending DeleteTablet for 3 replicas of tablet a6347e081ff14b9789c4f476ad9816ec
I20260619 14:48:25.706506 12022 tablet_service.cc:1514] Processing DeleteTablet for tablet a6347e081ff14b9789c4f476ad9816ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:48:25 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41196
I20260619 14:48:25.707082 11759 tablet_service.cc:1514] Processing DeleteTablet for tablet a6347e081ff14b9789c4f476ad9816ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:48:25 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48564
I20260619 14:48:25.708398 11905 tablet_service.cc:1514] Processing DeleteTablet for tablet a6347e081ff14b9789c4f476ad9816ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:48:25 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42526
I20260619 14:48:25.710139 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 11703
I20260619 14:48:25.720995 12151 tablet_replica.cc:330] T a6347e081ff14b9789c4f476ad9816ec P 9c0117c4102847768af3663e2fe5e3b3: stopping tablet replica
I20260619 14:48:25.721487 12151 raft_consensus.cc:2229] T a6347e081ff14b9789c4f476ad9816ec P 9c0117c4102847768af3663e2fe5e3b3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:48:25.721513 12149 tablet_replica.cc:330] T a6347e081ff14b9789c4f476ad9816ec P 736cc5237b2d4787a868ba6f7e582985: stopping tablet replica
I20260619 14:48:25.721944 12151 raft_consensus.cc:2258] T a6347e081ff14b9789c4f476ad9816ec P 9c0117c4102847768af3663e2fe5e3b3 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260619 14:48:25.722713 11606 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46149 (error 108)
I20260619 14:48:25.723745 12151 ts_tablet_manager.cc:1905] T a6347e081ff14b9789c4f476ad9816ec P 9c0117c4102847768af3663e2fe5e3b3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:48:25.723886 12149 raft_consensus.cc:2229] T a6347e081ff14b9789c4f476ad9816ec P 736cc5237b2d4787a868ba6f7e582985 [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:48:25.724671 12149 raft_consensus.cc:2258] T a6347e081ff14b9789c4f476ad9816ec P 736cc5237b2d4787a868ba6f7e582985 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:48:25.724828 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 11842
I20260619 14:48:25.726694 12121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:48:25.726367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46149 (local address 127.0.0.1:48572)
0619 14:48:25.726512 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0619 14:48:25.726595 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260619 14:48:25.726924 11606 catalog_manager.cc:4584] TS b9e2526d02304506ba61d5770efdfcaa (127.0.98.1:46149): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a6347e081ff14b9789c4f476ad9816ec: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46149: connect: Connection refused (error 111)
W20260619 14:48:25.732013 12121 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:48:25.730649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40985 (local address 127.0.0.1:42540)
0619 14:48:25.730884 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0619 14:48:25.730901 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 14:48:25.731001 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:48:25.731896 (+ 895us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:40985: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":130}
I20260619 14:48:25.732600 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 11977
I20260619 14:48:25.742144 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 11590
I20260619 14:48:25.742179 12121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:48:25.741800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37843 (local address 127.0.0.1:41204)
0619 14:48:25.741991 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0619 14:48:25.742072 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.3:37843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260619 14:48:25.751899 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 14:48:26.109588 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 14:48:26.109736 392 mini_ranger.cc:66] Stopping Ranger...
I20260619 14:48:26.456902 392 mini_ranger.cc:68] Stopped Ranger
2026-06-19 14:48:26.457 UTC [11176] LOG: received smart shutdown request
2026-06-19 14:48:26.461 UTC [11176] LOG: background worker "logical replication launcher" (PID 11185) exited with exit code 1
2026-06-19 14:48:26.462 UTC [11180] LOG: shutting down
2026-06-19 14:48:26.723 UTC [11176] LOG: database system is shut down
2026-06-19T14:48:26Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (57213 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:48:26 dist-test-slave-2sb1 krb5kdc[12162](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:48:26 dist-test-slave-2sb1 krb5kdc[12162](info): set up 2 sockets
Jun 19 14:48:26 dist-test-slave-2sb1 krb5kdc[12162](info): commencing operation
krb5kdc: starting...
W20260619 14:48:27.997344 392 mini_kdc.cc:121] Time spent starting KDC: real 1.121s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/krb5kdc/test-user.keytab.
Jun 19 14:48:28 dist-test-slave-2sb1 krb5kdc[12162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880508, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:48:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:48:28Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260619 14:48:28.196429 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskYPr0M2/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/postgres -l logfile start
2026-06-19 14:48:31.467 UTC [12192] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-19 14:48:31.467 UTC [12192] LOG: listening on IPv4 address "127.0.98.20", port 40967
2026-06-19 14:48:31.472 UTC [12192] LOG: listening on Unix socket "/tmp/.s.PGSQL.40967"
2026-06-19 14:48:31.509 UTC [12195] LOG: database system was shut down at 2026-06-19 14:48:29 UTC
2026-06-19 14:48:31.514 UTC [12192] LOG: database system is ready to accept connections
I20260619 14:48:32.536871 392 mini_postgres.cc:96] Postgres bound to 40967
2026-06-19 14:48:32.544 UTC [12203] FATAL: database "slave" does not exist
127.0.98.20:40967 - accepting connections
I20260619 14:48:32.544977 392 mini_ranger.cc:161] Starting Ranger...
I20260619 14:48:32.563117 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260619 14:48:33.266099 392 mini_ranger.cc:87] Created ranger Postgres database
I20260619 14:48:33.266189 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ranger-admin
2026-06-19 14:48:33,573 [I] DB FLAVOR :POSTGRES
2026-06-19 14:48:33,574 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 14:48:33,574 [I] Checking connection..
2026-06-19 14:48:33,574 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYPr0M2/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYPr0M2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-19 14:48:33,842 [I] Checking connection passed.
2026-06-19 14:48:33,842 [I] --------- Verifying version history table ---------
2026-06-19 14:48:33,842 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYPr0M2/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYPr0M2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 14:48:34,139 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 14:48:34,140 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYPr0M2/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYPr0M2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 14:48:34,443 [I] Table x_db_version_h does not exist in database ranger
2026-06-19 14:48:34,444 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-19 14:48:34,444 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYPr0M2/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYPr0M2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40967/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskYPr0M2/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-19 14:48:34.726 UTC [12267] WARNING: there is no transaction in progress
2026-06-19 14:48:34,734 [I] create_dbversion_catalog.sql file imported successfully
2026-06-19 14:48:34,734 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYPr0M2/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYPr0M2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-19 14:48:35,008 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-19 14:48:35,009 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 14:48:35,009 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYPr0M2/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYPr0M2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-19 14:48:35,285 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYPr0M2/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYPr0M2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-19 14:48:35,564 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYPr0M2/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYPr0M2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-2sb1.c.gcp-upstream.internal','N') ;"
2026-06-19 14:48:35,834 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYPr0M2/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYPr0M2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 14:48:36,116 [I] Table x_portal_user does not exist in database ranger
2026-06-19 14:48:36,117 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYPr0M2/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYPr0M2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 14:48:36,392 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-19 14:48:36,393 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYPr0M2/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYPr0M2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 14:48:36,672 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYPr0M2/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYPr0M2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 14:48:36,941 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-19 14:48:36,942 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYPr0M2/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYPr0M2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40967/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskYPr0M2/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-19 14:48:38.207 UTC [12375] WARNING: there is no transaction in progress
2026-06-19 14:48:38.231 UTC [12375] WARNING: there is no transaction in progress
2026-06-19 14:48:38.253 UTC [12375] WARNING: there is no transaction in progress
2026-06-19 14:48:38.274 UTC [12375] WARNING: there is no transaction in progress
2026-06-19 14:48:38.298 UTC [12375] WARNING: there is no transaction in progress
2026-06-19 14:48:38.324 UTC [12375] WARNING: there is no transaction in progress
2026-06-19 14:48:38.399 UTC [12375] WARNING: there is no transaction in progress
2026-06-19 14:48:38.409 UTC [12375] WARNING: there is no transaction in progress
2026-06-19 14:48:38.428 UTC [12375] WARNING: there is no transaction in progress
2026-06-19 14:48:38.442 UTC [12375] WARNING: there is no transaction in progress
2026-06-19 14:48:38.456 UTC [12375] WARNING: there is no transaction in progress
2026-06-19 14:48:38.474 UTC [12375] WARNING: there is no transaction in progress
2026-06-19 14:48:38.488 UTC [12375] WARNING: there is no transaction in progress
2026-06-19 14:48:38.496 UTC [12375] WARNING: there is no transaction in progress
2026-06-19 14:48:38.506 UTC [12375] WARNING: there is no transaction in progress
2026-06-19 14:48:38.516 UTC [12375] WARNING: there is no transaction in progress
2026-06-19 14:48:39,463 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-19 14:48:39,464 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYPr0M2/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYPr0M2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-19 14:48:39,731 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-19 14:48:39,732 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYPr0M2/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYPr0M2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-19 14:48:40,030 [I] Table x_portal_user already exists in database 'ranger'
2026-06-19 14:48:40,031 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYPr0M2/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYPr0M2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-19 14:48:40,322 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-19 14:48:40,323 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYPr0M2/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYPr0M2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-19 14:48:40,621 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYPr0M2/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYPr0M2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-19 14:48:40,886 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYPr0M2/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYPr0M2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-2sb1.c.gcp-upstream.internal';"
2026-06-19 14:48:41,153 [I] CORE_DB_SCHEMA import status has been updated
2026-06-19 14:48:41,153 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYPr0M2/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYPr0M2/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40967/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-19 14:48:41,420 [I] DB_PATCHES have already been applied
I20260619 14:48:41.426296 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskYPr0M2/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskYPr0M2/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskYPr0M2/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYPr0M2/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260619 14:48:41.426374 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260619 14:48:41.430119 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:44855
Jun 19, 2026 2:48:41 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 2:48:41 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 2:48:41 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ranger-admin/ews/webapp
Jun 19, 2026 2:48:41 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 19, 2026 2:48:41 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 19, 2026 2:48:42 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ranger-admin/ews/webapp].
Jun 19, 2026 2:48:42 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260619 14:48:42.602067 392 mini_ranger.cc:160] Time spent starting Ranger: real 10.057s user 0.008s sys 0.003s
Jun 19, 2026 2:49:05 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 19, 2026 2:49:05 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 19, 2026 2:49:05 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 19, 2026 2:49:05 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 19, 2026 2:49:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 19, 2026 2:49:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 19, 2026 2:49:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 19, 2026 2:49:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 19, 2026 2:49:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 19, 2026 2:49:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 19, 2026 2:49:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 19, 2026 2:49:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 19, 2026 2:49:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 19, 2026 2:49:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 19, 2026 2:49:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 19, 2026 2:49:05 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 19, 2026 2:49:05 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 19, 2026 2:49:06 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260619 14:49:07.089792 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 19 14:49:07 dist-test-slave-2sb1 krb5kdc[12162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880547, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260619 14:49:07.187383 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260619 14:49:07.198928 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260619 14:49:08.431650 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260619 14:49:08.431728 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ranger-kms
I20260619 14:49:08.431741 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:40967
2026-06-19 14:49:08,614 [I] DB FLAVOR :POSTGRES
2026-06-19 14:49:08,614 [I] --------- Verifying Ranger DB connection ---------
2026-06-19 14:49:08,614 [I] Checking connection
2026-06-19 14:49:08,614 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYPr0M2/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40967/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-19 14:49:08,872 [I] connection success
2026-06-19 14:49:08,872 [I] --------- Verifying Ranger DB tables ---------
2026-06-19 14:49:08,872 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-19 14:49:08,873 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYPr0M2/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40967/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-19 14:49:09,151 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-19 14:49:09,151 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-19 14:49:09,151 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-19 14:49:09,152 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskYPr0M2/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:40967/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-19 14:49:09,447 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260619 14:49:09.969448 392 mini_ranger_kms.cc:310] Created kms service
I20260619 14:49:10.194157 392 mini_ranger_kms.cc:326] Created kudu user
I20260619 14:49:10.243222 392 mini_ranger_kms.cc:343] Created rangerkms user
Jun 19 14:49:10 dist-test-slave-2sb1 krb5kdc[12162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880547, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 14:49:10.682331 392 mini_ranger_kms.cc:384] Added ranger policy
I20260619 14:49:10.682499 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskYPr0M2/build/debug/bin/postgresql.jar:/tmp/dist-test-taskYPr0M2/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskYPr0M2/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260619 14:49:10.682534 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260619 14:49:10.686527 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12556
I20260619 14:49:10.686628 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:37319
Jun 19, 2026 2:49:11 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 19, 2026 2:49:11 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 19, 2026 2:49:11 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 19, 2026 2:49:11 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 19, 2026 2:49:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ranger-kms/ews/webapp].
Jun 19, 2026 2:49:12 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Jun 19 14:49:13 dist-test-slave-2sb1 krb5kdc[12162](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781880553, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:49:13.119541 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.932s user 0.000s sys 0.162s
I20260619 14:49:13.119729 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260619 14:49:13.119812 392 mini_ranger_kms.cc:401] 127.0.98.20:37319/kms/v1/keys
Jun 19 14:49:13 dist-test-slave-2sb1 krb5kdc[12162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880553, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 19 14:49:14 dist-test-slave-2sb1 krb5kdc[12162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880553, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 19 14:49:16 dist-test-slave-2sb1 krb5kdc[12162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880556, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 14:49:16.549924 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34309
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34309
--ranger_config_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:37319/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:49:16.689294 12606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:49:16.689850 12606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:49:16.689960 12606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:49:16.697762 12606 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:49:16.697849 12606 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:49:16.697872 12606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:49:16.697891 12606 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:49:16.697953 12606 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:49:16.704877 12606 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:37319/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34309
--ranger_config_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34309
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12606
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:49:16.706128 12606 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:49:16.707281 12606 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:49:16.720098 12612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:49:16.720098 12614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:49:16.720078 12611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:49:16.720745 12606 server_base.cc:1034] running on GCE node
I20260619 14:49:16.721432 12606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:49:16.722798 12606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:49:16.724094 12606 hybrid_clock.cc:648] HybridClock initialized: now 1781880556724063 us; error 53 us; skew 500 ppm
Jun 19 14:49:16 dist-test-slave-2sb1 krb5kdc[12162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880556, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:49:16.728873 12606 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 14:49:16.730243 12606 webserver.cc:460] Webserver started at http://127.0.98.62:34793/ using document root <none> and password file <none>
I20260619 14:49:16.730876 12606 fs_manager.cc:359] Metadata directory not provided
I20260619 14:49:16.730964 12606 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:49:16.731194 12606 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 14:49:16 dist-test-slave-2sb1 krb5kdc[12162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880556, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 14:49:16.790086 12606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/master-0/data/instance:
uuid: "cb5d50f6090d4e8a8c49946022365fcb"
format_stamp: "Formatted at 2026-06-19 14:49:16 on dist-test-slave-2sb1"
server_key: "c70e2d8e7f88a54a261514de41710b1b"
server_key_iv: "479d7331fc2c9424928e612ccbbc0c6e"
server_key_version: "kuduclusterkey@0"
I20260619 14:49:16.790846 12606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/master-0/wal/instance:
uuid: "cb5d50f6090d4e8a8c49946022365fcb"
format_stamp: "Formatted at 2026-06-19 14:49:16 on dist-test-slave-2sb1"
server_key: "c70e2d8e7f88a54a261514de41710b1b"
server_key_iv: "479d7331fc2c9424928e612ccbbc0c6e"
server_key_version: "kuduclusterkey@0"
I20260619 14:49:16.796173 12606 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260619 14:49:16.812618 12621 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:16.814248 12606 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260619 14:49:16.814375 12606 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/master-0/wal
uuid: "cb5d50f6090d4e8a8c49946022365fcb"
format_stamp: "Formatted at 2026-06-19 14:49:16 on dist-test-slave-2sb1"
server_key: "c70e2d8e7f88a54a261514de41710b1b"
server_key_iv: "479d7331fc2c9424928e612ccbbc0c6e"
server_key_version: "kuduclusterkey@0"
I20260619 14:49:16.814538 12606 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:49:16.842381 12606 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:49:16.847148 12606 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:49:16.847406 12606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:49:16.856350 12606 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34309
I20260619 14:49:16.856365 12673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34309 every 8 connection(s)
I20260619 14:49:16.857472 12606 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:49:16.860562 12674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:49:16.861207 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 12606
I20260619 14:49:16.861332 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/master-0/wal/instance
I20260619 14:49:16.861644 392 external_mini_cluster.cc:1428] Setting key ed2407a455a28f600c3f3ef46b5b2131
I20260619 14:49:16.868994 12674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb5d50f6090d4e8a8c49946022365fcb: Bootstrap starting.
I20260619 14:49:16.872820 12674 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb5d50f6090d4e8a8c49946022365fcb: Neither blocks nor log segments found. Creating new log.
Jun 19 14:49:16 dist-test-slave-2sb1 krb5kdc[12162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880556, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:49:16.873835 12674 log.cc:824] T 00000000000000000000000000000000 P cb5d50f6090d4e8a8c49946022365fcb: Log is configured to *not* fsync() on all Append() calls
I20260619 14:49:16.877380 12674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb5d50f6090d4e8a8c49946022365fcb: No bootstrap required, opened a new log
I20260619 14:49:16.882088 12674 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb5d50f6090d4e8a8c49946022365fcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb5d50f6090d4e8a8c49946022365fcb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34309 } }
I20260619 14:49:16.882401 12674 raft_consensus.cc:374] T 00000000000000000000000000000000 P cb5d50f6090d4e8a8c49946022365fcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:49:16.882493 12674 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb5d50f6090d4e8a8c49946022365fcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb5d50f6090d4e8a8c49946022365fcb, State: Initialized, Role: FOLLOWER
I20260619 14:49:16.883047 12674 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb5d50f6090d4e8a8c49946022365fcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb5d50f6090d4e8a8c49946022365fcb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34309 } }
I20260619 14:49:16.883220 12674 raft_consensus.cc:388] T 00000000000000000000000000000000 P cb5d50f6090d4e8a8c49946022365fcb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:49:16.883285 12677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:16.863262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35934 (local address 127.0.98.62:34309)
0619 14:49:16.863792 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:16.863815 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:16.863863 (+ 48us) server_negotiation.cc:407] Connection header received
0619 14:49:16.864748 (+ 885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:16.864775 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:16.865235 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:16.865630 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:49:16.866585 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:16.868110 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:16.869925 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:16.870817 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:16.871073 (+ 256us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:16.874415 (+ 3342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:49:16.874424 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:49:16.874454 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:49:16.874524 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:49:16.878046 (+ 3522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:16.879014 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:16.879033 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:16.879041 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:16.879198 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:16.879559 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:16.879562 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:16.879564 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:16.880206 (+ 642us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:49:16.881099 (+ 893us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:16.881274 (+ 175us) server_negotiation.cc:299] Negotiation successful
0619 14:49:16.881611 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":165,"threads_started":1}
I20260619 14:49:16.883306 12674 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb5d50f6090d4e8a8c49946022365fcb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:49:16.883486 12674 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb5d50f6090d4e8a8c49946022365fcb [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:49:16.884557 12674 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb5d50f6090d4e8a8c49946022365fcb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb5d50f6090d4e8a8c49946022365fcb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34309 } }
I20260619 14:49:16.885035 12674 leader_election.cc:302] T 00000000000000000000000000000000 P cb5d50f6090d4e8a8c49946022365fcb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb5d50f6090d4e8a8c49946022365fcb; no voters:
I20260619 14:49:16.885404 12674 leader_election.cc:288] T 00000000000000000000000000000000 P cb5d50f6090d4e8a8c49946022365fcb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:49:16.885541 12679 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb5d50f6090d4e8a8c49946022365fcb [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:49:16.885789 12679 raft_consensus.cc:686] T 00000000000000000000000000000000 P cb5d50f6090d4e8a8c49946022365fcb [term 1 LEADER]: Becoming Leader. State: Replica: cb5d50f6090d4e8a8c49946022365fcb, State: Running, Role: LEADER
I20260619 14:49:16.886186 12679 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb5d50f6090d4e8a8c49946022365fcb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb5d50f6090d4e8a8c49946022365fcb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34309 } }
I20260619 14:49:16.886818 12674 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb5d50f6090d4e8a8c49946022365fcb [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:49:16.887663 12680 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb5d50f6090d4e8a8c49946022365fcb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cb5d50f6090d4e8a8c49946022365fcb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb5d50f6090d4e8a8c49946022365fcb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34309 } } }
I20260619 14:49:16.887669 12679 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb5d50f6090d4e8a8c49946022365fcb [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb5d50f6090d4e8a8c49946022365fcb. Latest consensus state: current_term: 1 leader_uuid: "cb5d50f6090d4e8a8c49946022365fcb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb5d50f6090d4e8a8c49946022365fcb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34309 } } }
I20260619 14:49:16.887856 12680 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb5d50f6090d4e8a8c49946022365fcb [sys.catalog]: This master's current role is: LEADER
I20260619 14:49:16.887856 12679 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb5d50f6090d4e8a8c49946022365fcb [sys.catalog]: This master's current role is: LEADER
I20260619 14:49:16.888291 12688 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:49:16.892061 12688 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:49:16.893107 12674 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 14:49:16.910544 12688 catalog_manager.cc:1347] Generated new cluster ID: ae8c4f25e16a48e1a768eaa268e72494
I20260619 14:49:16.910677 12688 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:49:16.939352 12688 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:49:16.942073 12688 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:49:16.950982 12688 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cb5d50f6090d4e8a8c49946022365fcb: Generated new TSK 0
I20260619 14:49:16.952162 12688 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 14:49:17.792588 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:37319/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34309
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:49:17.977116 12719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:49:17.977617 12719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:49:17.977730 12719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:49:17.984920 12719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:49:17.985105 12719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:49:17.985257 12719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 14:49:17.993852 12719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:37319/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34309
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:49:17.995357 12719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:49:17.996724 12719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:49:18.011134 12727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:49:18.011564 12729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:49:18.012243 12719 server_base.cc:1034] running on GCE node
W20260619 14:49:18.012813 12726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:49:18.013533 12719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:49:18.014436 12719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:49:18.021890 12719 hybrid_clock.cc:648] HybridClock initialized: now 1781880558021793 us; error 111 us; skew 500 ppm
Jun 19 14:49:18 dist-test-slave-2sb1 krb5kdc[12162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880558, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:49:18.026175 12719 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 14:49:18.027388 12719 webserver.cc:460] Webserver started at http://127.0.98.1:39195/ using document root <none> and password file <none>
I20260619 14:49:18.028133 12719 fs_manager.cc:359] Metadata directory not provided
I20260619 14:49:18.028229 12719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:49:18.028455 12719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 14:49:18 dist-test-slave-2sb1 krb5kdc[12162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880558, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 14:49:18.049532 12719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ts-0/data/instance:
uuid: "6e86100f05cb49eea64fb896fc8d484d"
format_stamp: "Formatted at 2026-06-19 14:49:18 on dist-test-slave-2sb1"
server_key: "ad563874dd19e43f4bd46da3448ec9ce"
server_key_iv: "f62ee2a5cace3578f4957c584b10f006"
server_key_version: "kuduclusterkey@0"
I20260619 14:49:18.052294 12719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ts-0/wal/instance:
uuid: "6e86100f05cb49eea64fb896fc8d484d"
format_stamp: "Formatted at 2026-06-19 14:49:18 on dist-test-slave-2sb1"
server_key: "ad563874dd19e43f4bd46da3448ec9ce"
server_key_iv: "f62ee2a5cace3578f4957c584b10f006"
server_key_version: "kuduclusterkey@0"
I20260619 14:49:18.060329 12719 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.000s
I20260619 14:49:18.081755 12736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:18.083878 12719 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260619 14:49:18.084162 12719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ts-0/wal
uuid: "6e86100f05cb49eea64fb896fc8d484d"
format_stamp: "Formatted at 2026-06-19 14:49:18 on dist-test-slave-2sb1"
server_key: "ad563874dd19e43f4bd46da3448ec9ce"
server_key_iv: "f62ee2a5cace3578f4957c584b10f006"
server_key_version: "kuduclusterkey@0"
I20260619 14:49:18.084425 12719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:49:18.114468 12719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:49:18.119230 12719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:49:18.119524 12719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:49:18.120472 12719 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:49:18.121629 12719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:49:18.121740 12719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:18.121884 12719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:49:18.121969 12719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:18.133137 12719 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34993
I20260619 14:49:18.134476 12719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ts-0/data/info.pb
I20260619 14:49:18.135497 12851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34993 every 8 connection(s)
I20260619 14:49:18.139902 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 12719
I20260619 14:49:18.140020 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ts-0/wal/instance
I20260619 14:49:18.140336 392 external_mini_cluster.cc:1428] Setting key 877c125ef733ce1561fe47896ea4e3e4
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Jun 19 14:49:18 dist-test-slave-2sb1 krb5kdc[12162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880558, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 14:49:18.175665 12855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:18.140843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39855 (local address 127.0.98.62:34309)
0619 14:49:18.144715 (+ 3872us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:18.144720 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:18.149897 (+ 5177us) server_negotiation.cc:407] Connection header received
0619 14:49:18.149956 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:18.149959 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:18.150021 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:18.150108 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:49:18.157063 (+ 6955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:18.158003 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:18.160158 (+ 2155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:18.160567 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:18.160699 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:18.168010 (+ 7311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:49:18.168015 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:49:18.168018 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:49:18.168055 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:49:18.170208 (+ 2153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:18.173263 (+ 3055us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:18.173269 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:18.173271 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:18.173414 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:18.173986 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:18.173990 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:18.173992 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:18.174259 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:49:18.174355 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:18.175194 (+ 839us) server_negotiation.cc:299] Negotiation successful
0619 14:49:18.175372 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3755,"thread_start_us":87,"threads_started":1}
I20260619 14:49:18.177897 12854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:18.138422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34309 (local address 127.0.98.1:39855)
0619 14:49:18.148219 (+ 9797us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:18.148304 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0619 14:49:18.149516 (+ 1212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:49:18.150461 (+ 945us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:49:18.150472 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:49:18.150861 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:49:18.151582 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:18.151604 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:18.158257 (+ 6653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:18.158262 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:18.159943 (+ 1681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:18.159963 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:18.160974 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:18.160979 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:18.161224 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:18.162087 (+ 863us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:49:18.162113 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:49:18.167716 (+ 5603us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:49:18.170439 (+ 2723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:49:18.170448 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:49:18.170472 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:49:18.170927 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:49:18.173580 (+ 2653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:49:18.173584 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:49:18.173586 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:49:18.173856 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:49:18.174438 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:49:18.174446 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:49:18.174876 (+ 430us) client_negotiation.cc:769] Sending connection context
0619 14:49:18.176210 (+ 1334us) client_negotiation.cc:240] Negotiation successful
0619 14:49:18.176735 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9508,"thread_start_us":178,"threads_started":1}
I20260619 14:49:18.179970 12852 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34309
I20260619 14:49:18.180241 12852 heartbeater.cc:464] Registering TS with master...
I20260619 14:49:18.180922 12852 heartbeater.cc:511] Master 127.0.98.62:34309 requested a full tablet report, sending...
I20260619 14:49:18.182967 12638 ts_manager.cc:194] Registered new tserver with Master: 6e86100f05cb49eea64fb896fc8d484d (127.0.98.1:34993)
I20260619 14:49:18.184528 12638 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:39855
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 14:49:18.204885 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:37319/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34309
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 14:49:18 dist-test-slave-2sb1 krb5kdc[12162](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781880558, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260619 14:49:18.373168 12858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:49:18.373464 12858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:49:18.373512 12858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:49:18.378388 12858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:49:18.378506 12858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:49:18.378603 12858 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 14:49:18.388314 12858 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:37319/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34309
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12858
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:49:18.389653 12858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:49:18.390769 12858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:49:18.404032 12864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:49:18.404464 12858 server_base.cc:1034] running on GCE node
W20260619 14:49:18.404191 12865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:49:18.404858 12867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:49:18.405751 12858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:49:18.406769 12858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:49:18.414372 12858 hybrid_clock.cc:648] HybridClock initialized: now 1781880558414302 us; error 98 us; skew 500 ppm
Jun 19 14:49:18 dist-test-slave-2sb1 krb5kdc[12162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880558, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:49:18.418674 12858 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 14:49:18.420003 12858 webserver.cc:460] Webserver started at http://127.0.98.2:38547/ using document root <none> and password file <none>
I20260619 14:49:18.420773 12858 fs_manager.cc:359] Metadata directory not provided
I20260619 14:49:18.420867 12858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:49:18.421134 12858 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 14:49:18 dist-test-slave-2sb1 krb5kdc[12162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880558, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 14:49:18.437320 12858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ts-1/data/instance:
uuid: "26305e9bb02641a4ace1ddf74f544f00"
format_stamp: "Formatted at 2026-06-19 14:49:18 on dist-test-slave-2sb1"
server_key: "f4fb5cfe65dc21b41398d752f53db572"
server_key_iv: "accda3dcc640888d2957431bcb1ded91"
server_key_version: "kuduclusterkey@0"
I20260619 14:49:18.438339 12858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ts-1/wal/instance:
uuid: "26305e9bb02641a4ace1ddf74f544f00"
format_stamp: "Formatted at 2026-06-19 14:49:18 on dist-test-slave-2sb1"
server_key: "f4fb5cfe65dc21b41398d752f53db572"
server_key_iv: "accda3dcc640888d2957431bcb1ded91"
server_key_version: "kuduclusterkey@0"
I20260619 14:49:18.443619 12858 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260619 14:49:18.489482 12874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:18.491596 12858 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.003s sys 0.001s
I20260619 14:49:18.491792 12858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ts-1/wal
uuid: "26305e9bb02641a4ace1ddf74f544f00"
format_stamp: "Formatted at 2026-06-19 14:49:18 on dist-test-slave-2sb1"
server_key: "f4fb5cfe65dc21b41398d752f53db572"
server_key_iv: "accda3dcc640888d2957431bcb1ded91"
server_key_version: "kuduclusterkey@0"
I20260619 14:49:18.491966 12858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:49:18.532230 12858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:49:18.537003 12858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:49:18.537259 12858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:49:18.537983 12858 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:49:18.549744 12858 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:49:18.549928 12858 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260619 14:49:18.550052 12858 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:49:18.550124 12858 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:18.561841 12858 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40879
I20260619 14:49:18.563516 12858 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ts-1/data/info.pb
I20260619 14:49:18.564312 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 12858
I20260619 14:49:18.564441 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ts-1/wal/instance
I20260619 14:49:18.565076 392 external_mini_cluster.cc:1428] Setting key ded176d44ff60b9e39b2fd78df179f58
I20260619 14:49:18.571574 12987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40879 every 8 connection(s)
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Jun 19 14:49:18 dist-test-slave-2sb1 krb5kdc[12162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880558, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:49:18.593011 12855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:18.576003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47253 (local address 127.0.98.62:34309)
0619 14:49:18.576420 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:18.576426 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:18.576685 (+ 259us) server_negotiation.cc:407] Connection header received
0619 14:49:18.578099 (+ 1414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:18.578103 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:18.578171 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:18.578230 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:49:18.580298 (+ 2068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:18.581146 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:18.583665 (+ 2519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:18.584034 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:18.584143 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:18.587880 (+ 3737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:49:18.587885 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:49:18.587888 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:49:18.587928 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:49:18.589966 (+ 2038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:18.590802 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:18.590806 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:18.590808 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:18.590909 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:18.591383 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:18.591387 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:18.591389 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:18.591635 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:49:18.591716 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:18.592605 (+ 889us) server_negotiation.cc:299] Negotiation successful
0619 14:49:18.592814 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260619 14:49:18.596365 12991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:18.574639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34309 (local address 127.0.98.2:47253)
0619 14:49:18.576455 (+ 1816us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:18.576521 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0619 14:49:18.577777 (+ 1256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:49:18.578744 (+ 967us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:49:18.578756 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:49:18.579169 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:49:18.580082 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:18.580110 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:18.581350 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:18.581354 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:18.583444 (+ 2090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:18.583469 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:18.584388 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:18.584391 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:18.584633 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:18.585674 (+ 1041us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:49:18.585698 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:49:18.587581 (+ 1883us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:49:18.590175 (+ 2594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:49:18.590184 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:49:18.590206 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:49:18.590657 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:49:18.591036 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:49:18.591039 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:49:18.591041 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:49:18.591261 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:49:18.591811 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:49:18.591819 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:49:18.592295 (+ 476us) client_negotiation.cc:769] Sending connection context
0619 14:49:18.593340 (+ 1045us) client_negotiation.cc:240] Negotiation successful
0619 14:49:18.594876 (+ 1536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1559,"thread_start_us":223,"threads_started":1}
I20260619 14:49:18.598563 12988 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34309
I20260619 14:49:18.598878 12988 heartbeater.cc:464] Registering TS with master...
I20260619 14:49:18.599546 12988 heartbeater.cc:511] Master 127.0.98.62:34309 requested a full tablet report, sending...
I20260619 14:49:18.600922 12638 ts_manager.cc:194] Registered new tserver with Master: 26305e9bb02641a4ace1ddf74f544f00 (127.0.98.2:40879)
I20260619 14:49:18.601711 12638 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47253
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 14:49:18.627164 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:37319/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34309
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:49:18.810983 12993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:49:18.811267 12993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:49:18.811314 12993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:49:18.816784 12993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:49:18.816934 12993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:49:18.817054 12993 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
Jun 19 14:49:18 dist-test-slave-2sb1 krb5kdc[12162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880558, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 14:49:18.827026 12993 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:37319/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34309
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12993
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:49:18.836123 12993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:49:18.844966 12993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:49:18.871677 12999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:49:18.872341 13001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:49:18.878712 12998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:49:18.883577 12993 server_base.cc:1034] running on GCE node
I20260619 14:49:18.884436 12993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:49:18.885571 12993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:49:18.896793 12993 hybrid_clock.cc:648] HybridClock initialized: now 1781880558896681 us; error 129 us; skew 500 ppm
Jun 19 14:49:18 dist-test-slave-2sb1 krb5kdc[12162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880558, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:49:18.901006 12993 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 14:49:18.902324 12993 webserver.cc:460] Webserver started at http://127.0.98.3:34667/ using document root <none> and password file <none>
I20260619 14:49:18.903152 12993 fs_manager.cc:359] Metadata directory not provided
I20260619 14:49:18.903302 12993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:49:18.903626 12993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 19 14:49:18 dist-test-slave-2sb1 krb5kdc[12162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880558, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 14:49:18.923923 12993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ts-2/data/instance:
uuid: "54040795b66642fcb61df15d9cf09111"
format_stamp: "Formatted at 2026-06-19 14:49:18 on dist-test-slave-2sb1"
server_key: "4aeb9ab181a97069bbe8a8f13c2351b0"
server_key_iv: "d72b31f0879237e94ec7345674cc9a08"
server_key_version: "kuduclusterkey@0"
I20260619 14:49:18.924796 12993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ts-2/wal/instance:
uuid: "54040795b66642fcb61df15d9cf09111"
format_stamp: "Formatted at 2026-06-19 14:49:18 on dist-test-slave-2sb1"
server_key: "4aeb9ab181a97069bbe8a8f13c2351b0"
server_key_iv: "d72b31f0879237e94ec7345674cc9a08"
server_key_version: "kuduclusterkey@0"
I20260619 14:49:18.942430 12993 fs_manager.cc:691] Time spent creating directory manager: real 0.017s user 0.004s sys 0.000s
I20260619 14:49:18.975663 13008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:18.977586 12993 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.003s sys 0.000s
I20260619 14:49:18.977774 12993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ts-2/wal
uuid: "54040795b66642fcb61df15d9cf09111"
format_stamp: "Formatted at 2026-06-19 14:49:18 on dist-test-slave-2sb1"
server_key: "4aeb9ab181a97069bbe8a8f13c2351b0"
server_key_iv: "d72b31f0879237e94ec7345674cc9a08"
server_key_version: "kuduclusterkey@0"
I20260619 14:49:18.977946 12993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:49:18.995354 12993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:49:18.999397 12993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:49:18.999603 12993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:49:19.000337 12993 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:49:19.001391 12993 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:49:19.001453 12993 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:19.001504 12993 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:49:19.001535 12993 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:19.014068 12993 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36461
I20260619 14:49:19.014506 13121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36461 every 8 connection(s)
I20260619 14:49:19.015447 12993 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ts-2/data/info.pb
I20260619 14:49:19.024183 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 12993
I20260619 14:49:19.024327 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ts-2/wal/instance
I20260619 14:49:19.024712 392 external_mini_cluster.cc:1428] Setting key 60c1b09bab835a4391c282db16097b9a
Jun 19 14:49:19 dist-test-slave-2sb1 krb5kdc[12162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880558, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:49:19.046866 12855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:19.024890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53381 (local address 127.0.98.62:34309)
0619 14:49:19.025063 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:19.025068 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:19.028341 (+ 3273us) server_negotiation.cc:407] Connection header received
0619 14:49:19.029883 (+ 1542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:19.029887 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:19.029951 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:19.030006 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:49:19.032291 (+ 2285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:19.033176 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:19.035496 (+ 2320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:19.035868 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:19.035965 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:19.042029 (+ 6064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:49:19.042034 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:49:19.042037 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:49:19.042071 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:49:19.044019 (+ 1948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:19.044852 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:19.044856 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:19.044858 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:19.044955 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:19.045406 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:19.045410 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:19.045412 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:19.045667 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:49:19.045755 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:19.046539 (+ 784us) server_negotiation.cc:299] Negotiation successful
0619 14:49:19.046692 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260619 14:49:19.049219 13124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:19.018953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34309 (local address 127.0.98.3:53381)
0619 14:49:19.028146 (+ 9193us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:19.028229 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0619 14:49:19.029580 (+ 1351us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:49:19.030526 (+ 946us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:49:19.030538 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:49:19.031083 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:49:19.032056 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:19.032082 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:19.033392 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:19.033395 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:19.035282 (+ 1887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:19.035299 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:19.036224 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:19.036228 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:19.036478 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:19.039666 (+ 3188us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:49:19.039696 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:49:19.041773 (+ 2077us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:49:19.044213 (+ 2440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:49:19.044220 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:49:19.044242 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:49:19.044709 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:49:19.045080 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:49:19.045083 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:49:19.045085 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:49:19.045289 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:49:19.045830 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:49:19.045838 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:49:19.046217 (+ 379us) client_negotiation.cc:769] Sending connection context
0619 14:49:19.047606 (+ 1389us) client_negotiation.cc:240] Negotiation successful
0619 14:49:19.048032 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8903,"thread_start_us":168,"threads_started":1}
I20260619 14:49:19.052663 13122 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34309
I20260619 14:49:19.052949 13122 heartbeater.cc:464] Registering TS with master...
I20260619 14:49:19.053592 13122 heartbeater.cc:511] Master 127.0.98.62:34309 requested a full tablet report, sending...
I20260619 14:49:19.054929 12638 ts_manager.cc:194] Registered new tserver with Master: 54040795b66642fcb61df15d9cf09111 (127.0.98.3:36461)
I20260619 14:49:19.055716 12638 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53381
I20260619 14:49:19.063627 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:49:19.064361 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 12719
I20260619 14:49:19.071951 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 12858
I20260619 14:49:19.081089 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 12993
I20260619 14:49:19.088717 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 12606
I20260619 14:49:19.098869 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34309
--webserver_interface=127.0.98.62
--webserver_port=34793
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34309
--ranger_config_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:37319/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:37319/kms with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:49:19.223912 13125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:49:19.224291 13125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:49:19.224370 13125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:49:19.229956 13125 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:49:19.230088 13125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:49:19.230134 13125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:49:19.230187 13125 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:49:19.230243 13125 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:49:19.236898 13125 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:37319/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34309
--ranger_config_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34309
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=34793
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13125
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:49:19.238133 13125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:49:19.239282 13125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:49:19.249022 13130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:49:19.249053 13131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:49:19.249013 13133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:49:19.249106 13125 server_base.cc:1034] running on GCE node
I20260619 14:49:19.250010 13125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:49:19.251288 13125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:49:19.252486 13125 hybrid_clock.cc:648] HybridClock initialized: now 1781880559252441 us; error 65 us; skew 500 ppm
Jun 19 14:49:19 dist-test-slave-2sb1 krb5kdc[12162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880559, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:49:19.256211 13125 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 14:49:19.257339 13125 webserver.cc:460] Webserver started at http://127.0.98.62:34793/ using document root <none> and password file <none>
I20260619 14:49:19.257982 13125 fs_manager.cc:359] Metadata directory not provided
I20260619 14:49:19.258090 13125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 14:49:19 dist-test-slave-2sb1 krb5kdc[12162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880559, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 14:49:19.343251 13125 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 14:49:19.346280 13141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:19.347988 13125 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 14:49:19.348193 13125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/master-0/wal
uuid: "cb5d50f6090d4e8a8c49946022365fcb"
format_stamp: "Formatted at 2026-06-19 14:49:16 on dist-test-slave-2sb1"
server_key: "c70e2d8e7f88a54a261514de41710b1b"
server_key_iv: "479d7331fc2c9424928e612ccbbc0c6e"
server_key_version: "kuduclusterkey@0"
I20260619 14:49:19.348982 13125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:49:19.391350 13125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:49:19.395324 13125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:49:19.395613 13125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:49:19.404349 13193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34309 every 8 connection(s)
I20260619 14:49:19.404345 13125 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34309
I20260619 14:49:19.405697 13125 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:49:19.410254 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 13125
I20260619 14:49:19.410794 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:37319/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:34993
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:34309
--webserver_port=39195
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:37319/kms with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:49:19.411445 13194 sys_catalog.cc:263] Verifying existing consensus state
I20260619 14:49:19.413939 13194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb5d50f6090d4e8a8c49946022365fcb: Bootstrap starting.
I20260619 14:49:19.432782 13194 log.cc:824] T 00000000000000000000000000000000 P cb5d50f6090d4e8a8c49946022365fcb: Log is configured to *not* fsync() on all Append() calls
I20260619 14:49:19.440389 13194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb5d50f6090d4e8a8c49946022365fcb: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 14:49:19.440925 13194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb5d50f6090d4e8a8c49946022365fcb: Bootstrap complete.
I20260619 14:49:19.445327 13194 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb5d50f6090d4e8a8c49946022365fcb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb5d50f6090d4e8a8c49946022365fcb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34309 } }
I20260619 14:49:19.445760 13194 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb5d50f6090d4e8a8c49946022365fcb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb5d50f6090d4e8a8c49946022365fcb, State: Initialized, Role: FOLLOWER
I20260619 14:49:19.446322 13194 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb5d50f6090d4e8a8c49946022365fcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb5d50f6090d4e8a8c49946022365fcb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34309 } }
I20260619 14:49:19.446445 13194 raft_consensus.cc:388] T 00000000000000000000000000000000 P cb5d50f6090d4e8a8c49946022365fcb [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:49:19.446534 13194 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb5d50f6090d4e8a8c49946022365fcb [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:49:19.446688 13194 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb5d50f6090d4e8a8c49946022365fcb [term 1 FOLLOWER]: Advancing to term 2
I20260619 14:49:19.448851 13194 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb5d50f6090d4e8a8c49946022365fcb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb5d50f6090d4e8a8c49946022365fcb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34309 } }
I20260619 14:49:19.449244 13194 leader_election.cc:302] T 00000000000000000000000000000000 P cb5d50f6090d4e8a8c49946022365fcb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb5d50f6090d4e8a8c49946022365fcb; no voters:
I20260619 14:49:19.449663 13194 leader_election.cc:288] T 00000000000000000000000000000000 P cb5d50f6090d4e8a8c49946022365fcb [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 14:49:19.449770 13198 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb5d50f6090d4e8a8c49946022365fcb [term 2 FOLLOWER]: Leader election won for term 2
I20260619 14:49:19.449947 13198 raft_consensus.cc:686] T 00000000000000000000000000000000 P cb5d50f6090d4e8a8c49946022365fcb [term 2 LEADER]: Becoming Leader. State: Replica: cb5d50f6090d4e8a8c49946022365fcb, State: Running, Role: LEADER
I20260619 14:49:19.450302 13198 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb5d50f6090d4e8a8c49946022365fcb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb5d50f6090d4e8a8c49946022365fcb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34309 } }
I20260619 14:49:19.451004 13194 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb5d50f6090d4e8a8c49946022365fcb [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:49:19.451303 13200 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb5d50f6090d4e8a8c49946022365fcb [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb5d50f6090d4e8a8c49946022365fcb. Latest consensus state: current_term: 2 leader_uuid: "cb5d50f6090d4e8a8c49946022365fcb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb5d50f6090d4e8a8c49946022365fcb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34309 } } }
I20260619 14:49:19.451440 13200 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb5d50f6090d4e8a8c49946022365fcb [sys.catalog]: This master's current role is: LEADER
I20260619 14:49:19.451679 13199 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb5d50f6090d4e8a8c49946022365fcb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "cb5d50f6090d4e8a8c49946022365fcb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb5d50f6090d4e8a8c49946022365fcb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34309 } } }
I20260619 14:49:19.451802 13199 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb5d50f6090d4e8a8c49946022365fcb [sys.catalog]: This master's current role is: LEADER
I20260619 14:49:19.451835 13202 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:49:19.456025 13202 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:49:19.456574 13194 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260619 14:49:19.456887 13202 catalog_manager.cc:1259] Loaded cluster ID: ae8c4f25e16a48e1a768eaa268e72494
I20260619 14:49:19.456944 13202 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:49:19.459546 13202 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:49:19.460354 13202 catalog_manager.cc:5879] T 00000000000000000000000000000000 P cb5d50f6090d4e8a8c49946022365fcb: Loaded TSK: 0
I20260619 14:49:19.461701 13202 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 14:49:19.573367 13196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:49:19.573773 13196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:49:19.573843 13196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:49:19.581077 13196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:49:19.581185 13196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:49:19.581403 13196 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 14:49:19.590114 13196 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:37319/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:34993
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=39195
--tserver_master_addrs=127.0.98.62:34309
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13196
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:49:19.591820 13196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:49:19.593511 13196 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:49:19.608243 13234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:49:19.608397 13231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:49:19.608300 13232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:49:19.608731 13196 server_base.cc:1034] running on GCE node
I20260619 14:49:19.609349 13196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:49:19.610411 13196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:49:19.611654 13196 hybrid_clock.cc:648] HybridClock initialized: now 1781880559611615 us; error 55 us; skew 500 ppm
Jun 19 14:49:19 dist-test-slave-2sb1 krb5kdc[12162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880559, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:49:19.615893 13196 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 14:49:19.617288 13196 webserver.cc:460] Webserver started at http://127.0.98.1:39195/ using document root <none> and password file <none>
I20260619 14:49:19.618070 13196 fs_manager.cc:359] Metadata directory not provided
I20260619 14:49:19.618185 13196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 14:49:19 dist-test-slave-2sb1 krb5kdc[12162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880559, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 14:49:19.643774 13196 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 14:49:19.647336 13242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:19.649201 13196 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260619 14:49:19.649367 13196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ts-0/wal
uuid: "6e86100f05cb49eea64fb896fc8d484d"
format_stamp: "Formatted at 2026-06-19 14:49:18 on dist-test-slave-2sb1"
server_key: "ad563874dd19e43f4bd46da3448ec9ce"
server_key_iv: "f62ee2a5cace3578f4957c584b10f006"
server_key_version: "kuduclusterkey@0"
I20260619 14:49:19.650208 13196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:49:19.678375 13196 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:49:19.682781 13196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:49:19.683068 13196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:49:19.683897 13196 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:49:19.685132 13196 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:49:19.685225 13196 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:19.685308 13196 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:49:19.685348 13196 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:19.697865 13196 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34993
I20260619 14:49:19.699052 13196 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ts-0/data/info.pb
I20260619 14:49:19.701035 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 13196
I20260619 14:49:19.701519 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:37319/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:40879
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:34309
--webserver_port=38547
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:37319/kms with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:49:19.705938 13355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34993 every 8 connection(s)
Jun 19 14:49:19 dist-test-slave-2sb1 krb5kdc[12162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880559, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:49:19.747555 13360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:19.714046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34309 (local address 127.0.98.1:44011)
0619 14:49:19.717158 (+ 3112us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:19.717244 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0619 14:49:19.718469 (+ 1225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:49:19.721357 (+ 2888us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:49:19.721367 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:49:19.721781 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:49:19.722601 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:19.722625 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:19.724773 (+ 2148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:19.724778 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:19.727163 (+ 2385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:19.727185 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:19.728962 (+ 1777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:19.728968 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:19.729208 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:19.730069 (+ 861us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:49:19.730094 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:49:19.738191 (+ 8097us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:49:19.742889 (+ 4698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:49:19.742897 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:49:19.742912 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:49:19.743422 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:49:19.744164 (+ 742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:49:19.744168 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:49:19.744171 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:49:19.744387 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:49:19.745310 (+ 923us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:49:19.745316 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:49:19.745673 (+ 357us) client_negotiation.cc:769] Sending connection context
0619 14:49:19.745893 (+ 220us) client_negotiation.cc:240] Negotiation successful
0619 14:49:19.746376 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2728,"thread_start_us":177,"threads_started":1}
I20260619 14:49:19.749996 13359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:19.713964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44011 (local address 127.0.98.62:34309)
0619 14:49:19.719221 (+ 5257us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:19.719255 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:19.719318 (+ 63us) server_negotiation.cc:407] Connection header received
0619 14:49:19.719784 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:19.719804 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:19.720317 (+ 513us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:19.720766 (+ 449us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:49:19.722852 (+ 2086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:19.724513 (+ 1661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:19.727420 (+ 2907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:19.728370 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:19.728600 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:19.738682 (+ 10082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:49:19.738688 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:49:19.738705 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:49:19.738901 (+ 196us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:49:19.742463 (+ 3562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:19.743765 (+ 1302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:19.743777 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:19.743784 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:19.743999 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:19.744532 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:19.744536 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:19.744538 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:19.745153 (+ 615us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:49:19.745310 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:19.748480 (+ 3170us) server_negotiation.cc:299] Negotiation successful
0619 14:49:19.748984 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4962,"thread_start_us":132,"threads_started":1}
I20260619 14:49:19.752987 13356 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34309
I20260619 14:49:19.753285 13356 heartbeater.cc:464] Registering TS with master...
I20260619 14:49:19.753919 13356 heartbeater.cc:511] Master 127.0.98.62:34309 requested a full tablet report, sending...
W20260619 14:49:19.756526 13356 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34309 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 14:49:19.756721 13356 heartbeater.cc:464] Registering TS with master...
I20260619 14:49:19.756943 13356 heartbeater.cc:511] Master 127.0.98.62:34309 requested a full tablet report, sending...
I20260619 14:49:19.757974 13356 heartbeater.cc:464] Registering TS with master...
I20260619 14:49:19.758191 13356 heartbeater.cc:511] Master 127.0.98.62:34309 requested a full tablet report, sending...
W20260619 14:49:19.759009 13356 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 14:49:19.894877 13358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:49:19.895166 13358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:49:19.895211 13358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:49:19.900813 13358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:49:19.900892 13358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:49:19.900976 13358 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 14:49:19.907130 13358 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:37319/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:40879
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=38547
--tserver_master_addrs=127.0.98.62:34309
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13358
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:49:19.908313 13358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:49:19.909564 13358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:49:19.919986 13368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:49:19.919986 13366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:49:19.920816 13365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:49:19.920956 13358 server_base.cc:1034] running on GCE node
I20260619 14:49:19.921432 13358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:49:19.922169 13358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:49:19.923375 13358 hybrid_clock.cc:648] HybridClock initialized: now 1781880559923350 us; error 43 us; skew 500 ppm
Jun 19 14:49:19 dist-test-slave-2sb1 krb5kdc[12162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880559, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:49:19.927079 13358 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 14:49:19.928229 13358 webserver.cc:460] Webserver started at http://127.0.98.2:38547/ using document root <none> and password file <none>
I20260619 14:49:19.928905 13358 fs_manager.cc:359] Metadata directory not provided
I20260619 14:49:19.929018 13358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 14:49:19 dist-test-slave-2sb1 krb5kdc[12162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880559, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 14:49:19.947154 13358 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:49:19.950088 13376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:19.951642 13358 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:49:19.951867 13358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ts-1/wal
uuid: "26305e9bb02641a4ace1ddf74f544f00"
format_stamp: "Formatted at 2026-06-19 14:49:18 on dist-test-slave-2sb1"
server_key: "f4fb5cfe65dc21b41398d752f53db572"
server_key_iv: "accda3dcc640888d2957431bcb1ded91"
server_key_version: "kuduclusterkey@0"
I20260619 14:49:19.952611 13358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:49:19.981516 13358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:49:19.985467 13358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:49:19.985698 13358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:49:19.986404 13358 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:49:19.987478 13358 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:49:19.987555 13358 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:19.987633 13358 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:49:19.987666 13358 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:19.999109 13358 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40879
I20260619 14:49:19.999136 13489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40879 every 8 connection(s)
I20260619 14:49:20.000253 13358 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ts-1/data/info.pb
I20260619 14:49:20.001551 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 13358
I20260619 14:49:20.002120 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:37319/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:36461
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:34309
--webserver_port=34667
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:37319/kms with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 14:49:20 dist-test-slave-2sb1 krb5kdc[12162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880559, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:49:20.023495 13359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:20.003594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50309 (local address 127.0.98.62:34309)
0619 14:49:20.003746 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:20.003750 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:20.006449 (+ 2699us) server_negotiation.cc:407] Connection header received
0619 14:49:20.007699 (+ 1250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:20.007703 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:20.007774 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:20.007891 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:49:20.010223 (+ 2332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:20.011077 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:20.013245 (+ 2168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:20.013622 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:20.013719 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:20.017241 (+ 3522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:49:20.017247 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:49:20.017251 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:49:20.017293 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:49:20.020113 (+ 2820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:20.021179 (+ 1066us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:20.021184 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:20.021186 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:20.021321 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:20.021920 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:20.021924 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:20.021925 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:20.022182 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:49:20.022274 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:20.023116 (+ 842us) server_negotiation.cc:299] Negotiation successful
0619 14:49:20.023291 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 14:49:20.025621 13493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:20.004194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34309 (local address 127.0.98.2:50309)
0619 14:49:20.006248 (+ 2054us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:20.006301 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0619 14:49:20.007367 (+ 1066us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:49:20.008234 (+ 867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:49:20.008243 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:49:20.009033 (+ 790us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:49:20.009964 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:20.009990 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:20.011367 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:20.011372 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:20.013036 (+ 1664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:20.013055 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:20.013992 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:20.013996 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:20.014224 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:20.015017 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:49:20.015040 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:49:20.016955 (+ 1915us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:49:20.020351 (+ 3396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:49:20.020361 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:49:20.020383 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:49:20.021013 (+ 630us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:49:20.021450 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:49:20.021453 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:49:20.021455 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:49:20.021663 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:49:20.022357 (+ 694us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:49:20.022365 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:49:20.022781 (+ 416us) client_negotiation.cc:769] Sending connection context
0619 14:49:20.024047 (+ 1266us) client_negotiation.cc:240] Negotiation successful
0619 14:49:20.024464 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1788,"thread_start_us":524,"threads_started":1}
I20260619 14:49:20.027561 13490 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34309
I20260619 14:49:20.027864 13490 heartbeater.cc:464] Registering TS with master...
I20260619 14:49:20.028534 13490 heartbeater.cc:511] Master 127.0.98.62:34309 requested a full tablet report, sending...
W20260619 14:49:20.030799 13490 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34309 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 14:49:20.031337 13490 heartbeater.cc:464] Registering TS with master...
I20260619 14:49:20.031562 13490 heartbeater.cc:511] Master 127.0.98.62:34309 requested a full tablet report, sending...
I20260619 14:49:20.033011 13490 heartbeater.cc:464] Registering TS with master...
I20260619 14:49:20.033257 13490 heartbeater.cc:511] Master 127.0.98.62:34309 requested a full tablet report, sending...
W20260619 14:49:20.034298 13490 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 14:49:20.172598 13492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:49:20.173102 13492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:49:20.173257 13492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:49:20.180723 13492 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:49:20.180831 13492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:49:20.180955 13492 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 14:49:20.189954 13492 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:37319/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:36461
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=34667
--tserver_master_addrs=127.0.98.62:34309
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13492
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:49:20.191293 13492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:49:20.193375 13492 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:49:20.207386 13500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:49:20.207379 13502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:49:20.207368 13499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:49:20.209167 13492 server_base.cc:1034] running on GCE node
I20260619 14:49:20.209880 13492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:49:20.211359 13492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:49:20.214684 13492 hybrid_clock.cc:648] HybridClock initialized: now 1781880560214578 us; error 128 us; skew 500 ppm
Jun 19 14:49:20 dist-test-slave-2sb1 krb5kdc[12162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880560, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:49:20.219036 13492 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 14:49:20.220502 13492 webserver.cc:460] Webserver started at http://127.0.98.3:34667/ using document root <none> and password file <none>
I20260619 14:49:20.221376 13492 fs_manager.cc:359] Metadata directory not provided
I20260619 14:49:20.221493 13492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 19 14:49:20 dist-test-slave-2sb1 krb5kdc[12162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880560, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 14:49:20.249778 13492 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 14:49:20.252830 13510 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:20.254503 13492 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:49:20.254664 13492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ts-2/wal
uuid: "54040795b66642fcb61df15d9cf09111"
format_stamp: "Formatted at 2026-06-19 14:49:18 on dist-test-slave-2sb1"
server_key: "4aeb9ab181a97069bbe8a8f13c2351b0"
server_key_iv: "d72b31f0879237e94ec7345674cc9a08"
server_key_version: "kuduclusterkey@0"
I20260619 14:49:20.255318 13492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:49:20.289530 13492 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:49:20.293915 13492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:49:20.294211 13492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:49:20.295116 13492 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:49:20.296299 13492 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:49:20.296375 13492 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:20.296465 13492 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:49:20.296518 13492 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:20.309625 13492 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36461
I20260619 14:49:20.309695 13623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36461 every 8 connection(s)
I20260619 14:49:20.311077 13492 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ts-2/data/info.pb
I20260619 14:49:20.311298 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 13492
I20260619 14:49:20.323603 13359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:20.312692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44758 (local address 127.0.98.62:34309)
0619 14:49:20.312850 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:20.312856 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:20.313181 (+ 325us) server_negotiation.cc:407] Connection header received
0619 14:49:20.313367 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:20.313373 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:20.313427 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:20.313545 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:49:20.314384 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:20.315140 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:20.316109 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:20.316601 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:20.316796 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:20.318824 (+ 2028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:49:20.318833 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:49:20.318840 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:49:20.318877 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:49:20.320951 (+ 2074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:20.321540 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:20.321545 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:20.321547 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:20.321642 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:20.321968 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:20.321971 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:20.321973 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:20.322248 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:49:20.322480 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:20.323278 (+ 798us) server_negotiation.cc:299] Negotiation successful
0619 14:49:20.323416 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
Jun 19 14:49:20 dist-test-slave-2sb1 krb5kdc[12162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880560, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:49:20.337978 13627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:20.313992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42967 (local address 127.0.98.62:34309)
0619 14:49:20.314231 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:20.314240 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:20.315244 (+ 1004us) server_negotiation.cc:407] Connection header received
0619 14:49:20.318672 (+ 3428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:20.318678 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:20.318767 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:20.318893 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:49:20.320905 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:20.321795 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:20.323747 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:20.324094 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:20.324205 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:20.331201 (+ 6996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:49:20.331206 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:49:20.331209 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:49:20.331246 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:49:20.333723 (+ 2477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:20.334577 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:20.334581 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:20.334583 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:20.334685 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:20.335168 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:20.335173 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:20.335184 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:20.335488 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:49:20.335631 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:20.337097 (+ 1466us) server_negotiation.cc:299] Negotiation successful
0619 14:49:20.337788 (+ 691us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":85,"threads_started":1}
I20260619 14:49:20.339599 13628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:20.314530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34309 (local address 127.0.98.3:42967)
0619 14:49:20.315058 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:20.315119 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0619 14:49:20.318203 (+ 3084us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:49:20.319279 (+ 1076us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:49:20.319292 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:49:20.319731 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:49:20.320685 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:20.320715 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:20.321949 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:20.321954 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:20.323571 (+ 1617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:20.323583 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:20.324425 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:20.324430 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:20.324705 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:20.328980 (+ 4275us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:49:20.329011 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:49:20.330878 (+ 1867us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:49:20.333930 (+ 3052us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:49:20.333939 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:49:20.333976 (+ 37us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:49:20.334421 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:49:20.334793 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:49:20.334796 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:49:20.334798 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:49:20.334998 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:49:20.335642 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:49:20.335650 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:49:20.336031 (+ 381us) client_negotiation.cc:769] Sending connection context
0619 14:49:20.336268 (+ 237us) client_negotiation.cc:240] Negotiation successful
0619 14:49:20.338480 (+ 2212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":122,"threads_started":1}
I20260619 14:49:20.340946 13624 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34309
I20260619 14:49:20.341244 13624 heartbeater.cc:464] Registering TS with master...
I20260619 14:49:20.341883 13624 heartbeater.cc:511] Master 127.0.98.62:34309 requested a full tablet report, sending...
W20260619 14:49:20.343858 13624 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34309 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260619 14:49:20.343972 13624 heartbeater.cc:464] Registering TS with master...
I20260619 14:49:20.344202 13624 heartbeater.cc:511] Master 127.0.98.62:34309 requested a full tablet report, sending...
I20260619 14:49:20.345497 13624 heartbeater.cc:464] Registering TS with master...
I20260619 14:49:20.345810 13624 heartbeater.cc:511] Master 127.0.98.62:34309 requested a full tablet report, sending...
W20260619 14:49:20.346917 13624 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 14:49:20.759248 13356 heartbeater.cc:464] Registering TS with master...
I20260619 14:49:20.759637 13356 heartbeater.cc:511] Master 127.0.98.62:34309 requested a full tablet report, sending...
I20260619 14:49:20.760991 13158 ts_manager.cc:194] Registered new tserver with Master: 6e86100f05cb49eea64fb896fc8d484d (127.0.98.1:34993)
I20260619 14:49:20.762305 13158 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:44011
Jun 19 14:49:20 dist-test-slave-2sb1 krb5kdc[12162](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781880560, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:49:21.034624 13490 heartbeater.cc:464] Registering TS with master...
I20260619 14:49:21.034996 13490 heartbeater.cc:511] Master 127.0.98.62:34309 requested a full tablet report, sending...
I20260619 14:49:21.035836 13158 ts_manager.cc:194] Registered new tserver with Master: 26305e9bb02641a4ace1ddf74f544f00 (127.0.98.2:40879)
I20260619 14:49:21.036489 13158 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50309
Jun 19 14:49:21 dist-test-slave-2sb1 krb5kdc[12162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880560, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 14:49:21.347211 13624 heartbeater.cc:464] Registering TS with master...
I20260619 14:49:21.347584 13624 heartbeater.cc:511] Master 127.0.98.62:34309 requested a full tablet report, sending...
I20260619 14:49:21.348721 13158 ts_manager.cc:194] Registered new tserver with Master: 54040795b66642fcb61df15d9cf09111 (127.0.98.3:36461)
I20260619 14:49:21.349537 13158 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42967
I20260619 14:49:21.355254 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:49:21.372393 13645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:21.357392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44760 (local address 127.0.98.62:34309)
0619 14:49:21.358413 (+ 1021us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:21.358418 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:21.358706 (+ 288us) server_negotiation.cc:407] Connection header received
0619 14:49:21.358954 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:21.358958 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:21.359015 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:21.359078 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:49:21.360205 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:21.361081 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:21.362313 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:21.362641 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:21.362727 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:21.367945 (+ 5218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:49:21.367950 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:49:21.367953 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:49:21.367993 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:49:21.370033 (+ 2040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:21.370844 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:21.370848 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:21.370850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:21.370957 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:21.371299 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:21.371303 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:21.371305 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:21.371543 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:49:21.371666 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:21.372056 (+ 390us) server_negotiation.cc:299] Negotiation successful
0619 14:49:21.372205 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":932,"thread_start_us":100,"threads_started":1}
I20260619 14:49:21.377866 13158 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44760:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:49:21.383414 13158 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:49:21.414899 13655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:21.397796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36461 (local address 127.0.0.1:54362)
0619 14:49:21.402894 (+ 5098us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:21.402968 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0619 14:49:21.403133 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:49:21.409721 (+ 6588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:49:21.409728 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:49:21.409755 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:49:21.409904 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:21.409910 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:21.411381 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:21.411385 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:21.412913 (+ 1528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:21.412928 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:21.414319 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:21.414323 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:21.414456 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:21.414492 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 14:49:21.414603 (+ 111us) client_negotiation.cc:240] Negotiation successful
0619 14:49:21.414742 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4784,"thread_start_us":129,"threads_started":1}
I20260619 14:49:21.416044 13651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:21.397796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54362 (local address 127.0.98.3:36461)
0619 14:49:21.408916 (+ 11120us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:21.408926 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:21.408960 (+ 34us) server_negotiation.cc:407] Connection header received
0619 14:49:21.409054 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:21.409060 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:21.409333 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:21.409505 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:49:21.410036 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:21.411223 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:21.413079 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:21.413975 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:21.414089 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:21.414255 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:21.415783 (+ 1528us) server_negotiation.cc:299] Negotiation successful
0619 14:49:21.415904 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10975,"thread_start_us":132,"threads_started":1}
I20260619 14:49:21.419821 13556 tablet_service.cc:1467] Processing CreateTablet for tablet 43cfa821fb584abeb2b6581aaa247693 (DEFAULT_TABLE table=test-table [id=f1679e7f20514cc482e8ade9f9521876]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:49:21.421280 13556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43cfa821fb584abeb2b6581aaa247693. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:49:21.423831 13653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:21.399303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34993 (local address 127.0.0.1:42956)
0619 14:49:21.403630 (+ 4327us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:21.403644 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:49:21.403737 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:49:21.418783 (+ 15046us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:49:21.418786 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:49:21.418811 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:49:21.418914 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:21.418920 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:21.420330 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:21.420334 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:21.421834 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:21.421849 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:21.423354 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:21.423359 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:21.423455 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:21.423483 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 14:49:21.423559 (+ 76us) client_negotiation.cc:240] Negotiation successful
0619 14:49:21.423670 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4252,"thread_start_us":118,"threads_started":1}
I20260619 14:49:21.424530 13654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:21.399833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42956 (local address 127.0.98.1:34993)
0619 14:49:21.404002 (+ 4169us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:21.404009 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:21.404037 (+ 28us) server_negotiation.cc:407] Connection header received
0619 14:49:21.404125 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:21.404131 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:21.418411 (+ 14280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:21.418596 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:49:21.419082 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:21.420181 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:21.422124 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:21.422993 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:21.423115 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:21.423277 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:21.424270 (+ 993us) server_negotiation.cc:299] Negotiation successful
0619 14:49:21.424390 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4043,"thread_start_us":85,"threads_started":1}
I20260619 14:49:21.428367 13290 tablet_service.cc:1467] Processing CreateTablet for tablet 43cfa821fb584abeb2b6581aaa247693 (DEFAULT_TABLE table=test-table [id=f1679e7f20514cc482e8ade9f9521876]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:49:21.429754 13290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43cfa821fb584abeb2b6581aaa247693. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:49:21.441044 13657 tablet_bootstrap.cc:492] T 43cfa821fb584abeb2b6581aaa247693 P 6e86100f05cb49eea64fb896fc8d484d: Bootstrap starting.
I20260619 14:49:21.442648 13655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:21.401460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40879 (local address 127.0.0.1:55076)
0619 14:49:21.415559 (+ 14099us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:21.415572 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:49:21.415652 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:49:21.428751 (+ 13099us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:49:21.428755 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:49:21.428784 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:49:21.428896 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:21.428904 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:21.430441 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:21.430444 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:21.431557 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:21.431572 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:21.442151 (+ 10579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:21.442155 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:21.442282 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:21.442315 (+ 33us) client_negotiation.cc:769] Sending connection context
0619 14:49:21.442392 (+ 77us) client_negotiation.cc:240] Negotiation successful
0619 14:49:21.442493 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14033,"thread_start_us":135,"threads_started":1}
I20260619 14:49:21.443168 13656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:21.401979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55076 (local address 127.0.98.2:40879)
0619 14:49:21.403469 (+ 1490us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:21.403477 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:21.417567 (+ 14090us) server_negotiation.cc:407] Connection header received
0619 14:49:21.417680 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:21.417687 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:21.417913 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:21.418077 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:49:21.429087 (+ 11010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:21.430257 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:21.440810 (+ 10553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:21.441748 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:21.441854 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:21.442004 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:21.442907 (+ 903us) server_negotiation.cc:299] Negotiation successful
0619 14:49:21.443028 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1367,"thread_start_us":71,"threads_started":1}
I20260619 14:49:21.446719 13424 tablet_service.cc:1467] Processing CreateTablet for tablet 43cfa821fb584abeb2b6581aaa247693 (DEFAULT_TABLE table=test-table [id=f1679e7f20514cc482e8ade9f9521876]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:49:21.447902 13424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43cfa821fb584abeb2b6581aaa247693. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:49:21.455628 13657 tablet_bootstrap.cc:654] T 43cfa821fb584abeb2b6581aaa247693 P 6e86100f05cb49eea64fb896fc8d484d: Neither blocks nor log segments found. Creating new log.
I20260619 14:49:21.457849 13657 log.cc:824] T 43cfa821fb584abeb2b6581aaa247693 P 6e86100f05cb49eea64fb896fc8d484d: Log is configured to *not* fsync() on all Append() calls
I20260619 14:49:21.460245 13660 tablet_bootstrap.cc:492] T 43cfa821fb584abeb2b6581aaa247693 P 26305e9bb02641a4ace1ddf74f544f00: Bootstrap starting.
I20260619 14:49:21.461387 13658 tablet_bootstrap.cc:492] T 43cfa821fb584abeb2b6581aaa247693 P 54040795b66642fcb61df15d9cf09111: Bootstrap starting.
I20260619 14:49:21.463048 13660 tablet_bootstrap.cc:654] T 43cfa821fb584abeb2b6581aaa247693 P 26305e9bb02641a4ace1ddf74f544f00: Neither blocks nor log segments found. Creating new log.
I20260619 14:49:21.464020 13660 log.cc:824] T 43cfa821fb584abeb2b6581aaa247693 P 26305e9bb02641a4ace1ddf74f544f00: Log is configured to *not* fsync() on all Append() calls
I20260619 14:49:21.465219 13657 tablet_bootstrap.cc:492] T 43cfa821fb584abeb2b6581aaa247693 P 6e86100f05cb49eea64fb896fc8d484d: No bootstrap required, opened a new log
I20260619 14:49:21.465652 13657 ts_tablet_manager.cc:1397] T 43cfa821fb584abeb2b6581aaa247693 P 6e86100f05cb49eea64fb896fc8d484d: Time spent bootstrapping tablet: real 0.025s user 0.006s sys 0.000s
I20260619 14:49:21.465934 13658 tablet_bootstrap.cc:654] T 43cfa821fb584abeb2b6581aaa247693 P 54040795b66642fcb61df15d9cf09111: Neither blocks nor log segments found. Creating new log.
I20260619 14:49:21.466905 13658 log.cc:824] T 43cfa821fb584abeb2b6581aaa247693 P 54040795b66642fcb61df15d9cf09111: Log is configured to *not* fsync() on all Append() calls
I20260619 14:49:21.470109 13657 raft_consensus.cc:348] T 43cfa821fb584abeb2b6581aaa247693 P 6e86100f05cb49eea64fb896fc8d484d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54040795b66642fcb61df15d9cf09111" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36461 } } peers { permanent_uuid: "6e86100f05cb49eea64fb896fc8d484d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34993 } } peers { permanent_uuid: "26305e9bb02641a4ace1ddf74f544f00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40879 } }
I20260619 14:49:21.470431 13657 raft_consensus.cc:374] T 43cfa821fb584abeb2b6581aaa247693 P 6e86100f05cb49eea64fb896fc8d484d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:49:21.470481 13657 raft_consensus.cc:729] T 43cfa821fb584abeb2b6581aaa247693 P 6e86100f05cb49eea64fb896fc8d484d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e86100f05cb49eea64fb896fc8d484d, State: Initialized, Role: FOLLOWER
I20260619 14:49:21.471047 13660 tablet_bootstrap.cc:492] T 43cfa821fb584abeb2b6581aaa247693 P 26305e9bb02641a4ace1ddf74f544f00: No bootstrap required, opened a new log
I20260619 14:49:21.470976 13657 consensus_queue.cc:260] T 43cfa821fb584abeb2b6581aaa247693 P 6e86100f05cb49eea64fb896fc8d484d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54040795b66642fcb61df15d9cf09111" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36461 } } peers { permanent_uuid: "6e86100f05cb49eea64fb896fc8d484d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34993 } } peers { permanent_uuid: "26305e9bb02641a4ace1ddf74f544f00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40879 } }
I20260619 14:49:21.471256 13660 ts_tablet_manager.cc:1397] T 43cfa821fb584abeb2b6581aaa247693 P 26305e9bb02641a4ace1ddf74f544f00: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20260619 14:49:21.472162 13657 ts_tablet_manager.cc:1428] T 43cfa821fb584abeb2b6581aaa247693 P 6e86100f05cb49eea64fb896fc8d484d: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260619 14:49:21.472550 13356 heartbeater.cc:503] Master 127.0.98.62:34309 was elected leader, sending a full tablet report...
I20260619 14:49:21.477488 13658 tablet_bootstrap.cc:492] T 43cfa821fb584abeb2b6581aaa247693 P 54040795b66642fcb61df15d9cf09111: No bootstrap required, opened a new log
I20260619 14:49:21.477871 13658 ts_tablet_manager.cc:1397] T 43cfa821fb584abeb2b6581aaa247693 P 54040795b66642fcb61df15d9cf09111: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.000s
I20260619 14:49:21.478977 13660 raft_consensus.cc:348] T 43cfa821fb584abeb2b6581aaa247693 P 26305e9bb02641a4ace1ddf74f544f00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54040795b66642fcb61df15d9cf09111" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36461 } } peers { permanent_uuid: "6e86100f05cb49eea64fb896fc8d484d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34993 } } peers { permanent_uuid: "26305e9bb02641a4ace1ddf74f544f00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40879 } }
I20260619 14:49:21.479379 13660 raft_consensus.cc:374] T 43cfa821fb584abeb2b6581aaa247693 P 26305e9bb02641a4ace1ddf74f544f00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:49:21.479476 13660 raft_consensus.cc:729] T 43cfa821fb584abeb2b6581aaa247693 P 26305e9bb02641a4ace1ddf74f544f00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26305e9bb02641a4ace1ddf74f544f00, State: Initialized, Role: FOLLOWER
I20260619 14:49:21.480095 13660 consensus_queue.cc:260] T 43cfa821fb584abeb2b6581aaa247693 P 26305e9bb02641a4ace1ddf74f544f00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54040795b66642fcb61df15d9cf09111" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36461 } } peers { permanent_uuid: "6e86100f05cb49eea64fb896fc8d484d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34993 } } peers { permanent_uuid: "26305e9bb02641a4ace1ddf74f544f00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40879 } }
I20260619 14:49:21.481412 13660 ts_tablet_manager.cc:1428] T 43cfa821fb584abeb2b6581aaa247693 P 26305e9bb02641a4ace1ddf74f544f00: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20260619 14:49:21.481881 13490 heartbeater.cc:503] Master 127.0.98.62:34309 was elected leader, sending a full tablet report...
I20260619 14:49:21.482414 13658 raft_consensus.cc:348] T 43cfa821fb584abeb2b6581aaa247693 P 54040795b66642fcb61df15d9cf09111 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54040795b66642fcb61df15d9cf09111" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36461 } } peers { permanent_uuid: "6e86100f05cb49eea64fb896fc8d484d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34993 } } peers { permanent_uuid: "26305e9bb02641a4ace1ddf74f544f00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40879 } }
I20260619 14:49:21.482699 13658 raft_consensus.cc:374] T 43cfa821fb584abeb2b6581aaa247693 P 54040795b66642fcb61df15d9cf09111 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:49:21.482765 13658 raft_consensus.cc:729] T 43cfa821fb584abeb2b6581aaa247693 P 54040795b66642fcb61df15d9cf09111 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54040795b66642fcb61df15d9cf09111, State: Initialized, Role: FOLLOWER
I20260619 14:49:21.483501 13658 consensus_queue.cc:260] T 43cfa821fb584abeb2b6581aaa247693 P 54040795b66642fcb61df15d9cf09111 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54040795b66642fcb61df15d9cf09111" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36461 } } peers { permanent_uuid: "6e86100f05cb49eea64fb896fc8d484d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34993 } } peers { permanent_uuid: "26305e9bb02641a4ace1ddf74f544f00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40879 } }
I20260619 14:49:21.485074 13658 ts_tablet_manager.cc:1428] T 43cfa821fb584abeb2b6581aaa247693 P 54040795b66642fcb61df15d9cf09111: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260619 14:49:21.485520 13624 heartbeater.cc:503] Master 127.0.98.62:34309 was elected leader, sending a full tablet report...
W20260619 14:49:21.503973 13491 tablet.cc:2369] T 43cfa821fb584abeb2b6581aaa247693 P 26305e9bb02641a4ace1ddf74f544f00: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 19 14:49:21 dist-test-slave-2sb1 krb5kdc[12162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880560, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260619 14:49:21.535807 13664 raft_consensus.cc:482] T 43cfa821fb584abeb2b6581aaa247693 P 26305e9bb02641a4ace1ddf74f544f00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:49:21.536042 13664 raft_consensus.cc:504] T 43cfa821fb584abeb2b6581aaa247693 P 26305e9bb02641a4ace1ddf74f544f00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54040795b66642fcb61df15d9cf09111" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36461 } } peers { permanent_uuid: "6e86100f05cb49eea64fb896fc8d484d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34993 } } peers { permanent_uuid: "26305e9bb02641a4ace1ddf74f544f00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40879 } }
I20260619 14:49:21.537542 13664 leader_election.cc:288] T 43cfa821fb584abeb2b6581aaa247693 P 26305e9bb02641a4ace1ddf74f544f00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 54040795b66642fcb61df15d9cf09111 (127.0.98.3:36461), 6e86100f05cb49eea64fb896fc8d484d (127.0.98.1:34993)
I20260619 14:49:21.545015 13654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:21.539052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33607 (local address 127.0.98.1:34993)
0619 14:49:21.539178 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:21.539183 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:21.539843 (+ 660us) server_negotiation.cc:407] Connection header received
0619 14:49:21.540026 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:21.540029 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:21.540079 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:21.540130 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:49:21.540531 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:21.541387 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:21.543501 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:21.544173 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:21.544265 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:21.544346 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:21.544776 (+ 430us) server_negotiation.cc:299] Negotiation successful
0619 14:49:21.544868 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 14:49:21.545758 13667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:21.538402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34993 (local address 127.0.98.2:33607)
0619 14:49:21.539767 (+ 1365us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:21.539779 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:49:21.539906 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:49:21.540269 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:49:21.540271 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:49:21.540288 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:49:21.540404 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:21.540409 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:21.541546 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:21.541549 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:21.543328 (+ 1779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:21.543344 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:21.544461 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:21.544465 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:21.544584 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:21.544608 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 14:49:21.545473 (+ 865us) client_negotiation.cc:240] Negotiation successful
0619 14:49:21.545561 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1305,"thread_start_us":98,"threads_started":1}
I20260619 14:49:21.546664 13310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43cfa821fb584abeb2b6581aaa247693" candidate_uuid: "26305e9bb02641a4ace1ddf74f544f00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e86100f05cb49eea64fb896fc8d484d" is_pre_election: true
I20260619 14:49:21.547020 13310 raft_consensus.cc:2454] T 43cfa821fb584abeb2b6581aaa247693 P 6e86100f05cb49eea64fb896fc8d484d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26305e9bb02641a4ace1ddf74f544f00 in term 0.
I20260619 14:49:21.547703 13378 leader_election.cc:302] T 43cfa821fb584abeb2b6581aaa247693 P 26305e9bb02641a4ace1ddf74f544f00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26305e9bb02641a4ace1ddf74f544f00, 6e86100f05cb49eea64fb896fc8d484d; no voters:
I20260619 14:49:21.547789 13651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:21.538892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33123 (local address 127.0.98.3:36461)
0619 14:49:21.539060 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:21.539066 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:21.539507 (+ 441us) server_negotiation.cc:407] Connection header received
0619 14:49:21.539668 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:21.539671 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:21.539752 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:21.539854 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:49:21.544126 (+ 4272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:21.544880 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:21.546404 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:21.547018 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:21.547105 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:21.547169 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:21.547537 (+ 368us) server_negotiation.cc:299] Negotiation successful
0619 14:49:21.547636 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 14:49:21.548048 13664 raft_consensus.cc:2781] T 43cfa821fb584abeb2b6581aaa247693 P 26305e9bb02641a4ace1ddf74f544f00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:49:21.548127 13664 raft_consensus.cc:482] T 43cfa821fb584abeb2b6581aaa247693 P 26305e9bb02641a4ace1ddf74f544f00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:49:21.548184 13664 raft_consensus.cc:3037] T 43cfa821fb584abeb2b6581aaa247693 P 26305e9bb02641a4ace1ddf74f544f00 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:49:21.549605 13664 raft_consensus.cc:504] T 43cfa821fb584abeb2b6581aaa247693 P 26305e9bb02641a4ace1ddf74f544f00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54040795b66642fcb61df15d9cf09111" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36461 } } peers { permanent_uuid: "6e86100f05cb49eea64fb896fc8d484d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34993 } } peers { permanent_uuid: "26305e9bb02641a4ace1ddf74f544f00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40879 } }
I20260619 14:49:21.550014 13664 leader_election.cc:288] T 43cfa821fb584abeb2b6581aaa247693 P 26305e9bb02641a4ace1ddf74f544f00 [CANDIDATE]: Term 1 election: Requested vote from peers 54040795b66642fcb61df15d9cf09111 (127.0.98.3:36461), 6e86100f05cb49eea64fb896fc8d484d (127.0.98.1:34993)
I20260619 14:49:21.551074 13310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43cfa821fb584abeb2b6581aaa247693" candidate_uuid: "26305e9bb02641a4ace1ddf74f544f00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e86100f05cb49eea64fb896fc8d484d"
I20260619 14:49:21.551221 13310 raft_consensus.cc:3037] T 43cfa821fb584abeb2b6581aaa247693 P 6e86100f05cb49eea64fb896fc8d484d [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:49:21.552569 13668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:21.537969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36461 (local address 127.0.98.2:33123)
0619 14:49:21.539409 (+ 1440us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:21.539425 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:49:21.539557 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:49:21.540001 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:49:21.540005 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:49:21.540025 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:49:21.540176 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:21.540183 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:21.545039 (+ 4856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:21.545042 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:21.546234 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:21.546247 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:21.547275 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:21.547278 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:21.547376 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:21.547401 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 14:49:21.548012 (+ 611us) client_negotiation.cc:240] Negotiation successful
0619 14:49:21.548102 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1367,"thread_start_us":125,"threads_started":1}
I20260619 14:49:21.552762 13310 raft_consensus.cc:2454] T 43cfa821fb584abeb2b6581aaa247693 P 6e86100f05cb49eea64fb896fc8d484d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26305e9bb02641a4ace1ddf74f544f00 in term 1.
I20260619 14:49:21.553766 13578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43cfa821fb584abeb2b6581aaa247693" candidate_uuid: "26305e9bb02641a4ace1ddf74f544f00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54040795b66642fcb61df15d9cf09111" is_pre_election: true
I20260619 14:49:21.554108 13578 raft_consensus.cc:2454] T 43cfa821fb584abeb2b6581aaa247693 P 54040795b66642fcb61df15d9cf09111 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26305e9bb02641a4ace1ddf74f544f00 in term 0.
I20260619 14:49:21.554478 13577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43cfa821fb584abeb2b6581aaa247693" candidate_uuid: "26305e9bb02641a4ace1ddf74f544f00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54040795b66642fcb61df15d9cf09111"
I20260619 14:49:21.554605 13577 raft_consensus.cc:3037] T 43cfa821fb584abeb2b6581aaa247693 P 54040795b66642fcb61df15d9cf09111 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:49:21.556047 13577 raft_consensus.cc:2454] T 43cfa821fb584abeb2b6581aaa247693 P 54040795b66642fcb61df15d9cf09111 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26305e9bb02641a4ace1ddf74f544f00 in term 1.
I20260619 14:49:21.557114 13377 leader_election.cc:302] T 43cfa821fb584abeb2b6581aaa247693 P 26305e9bb02641a4ace1ddf74f544f00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26305e9bb02641a4ace1ddf74f544f00, 54040795b66642fcb61df15d9cf09111; no voters:
I20260619 14:49:21.557787 13664 raft_consensus.cc:2781] T 43cfa821fb584abeb2b6581aaa247693 P 26305e9bb02641a4ace1ddf74f544f00 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:49:21.558018 13664 raft_consensus.cc:686] T 43cfa821fb584abeb2b6581aaa247693 P 26305e9bb02641a4ace1ddf74f544f00 [term 1 LEADER]: Becoming Leader. State: Replica: 26305e9bb02641a4ace1ddf74f544f00, State: Running, Role: LEADER
I20260619 14:49:21.558435 13664 consensus_queue.cc:237] T 43cfa821fb584abeb2b6581aaa247693 P 26305e9bb02641a4ace1ddf74f544f00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54040795b66642fcb61df15d9cf09111" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36461 } } peers { permanent_uuid: "6e86100f05cb49eea64fb896fc8d484d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34993 } } peers { permanent_uuid: "26305e9bb02641a4ace1ddf74f544f00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40879 } }
I20260619 14:49:21.562866 13158 catalog_manager.cc:5495] T 43cfa821fb584abeb2b6581aaa247693 P 26305e9bb02641a4ace1ddf74f544f00 reported cstate change: term changed from 0 to 1, leader changed from <none> to 26305e9bb02641a4ace1ddf74f544f00 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "26305e9bb02641a4ace1ddf74f544f00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "54040795b66642fcb61df15d9cf09111" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36461 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e86100f05cb49eea64fb896fc8d484d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26305e9bb02641a4ace1ddf74f544f00" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40879 } health_report { overall_health: HEALTHY } } }
W20260619 14:49:21.564013 13625 tablet.cc:2369] T 43cfa821fb584abeb2b6581aaa247693 P 54040795b66642fcb61df15d9cf09111: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:49:21.598295 13656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:21.592961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55088 (local address 127.0.98.2:40879)
0619 14:49:21.593376 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:21.593381 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:21.593394 (+ 13us) server_negotiation.cc:407] Connection header received
0619 14:49:21.593492 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:21.593495 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:21.593564 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:21.593680 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:49:21.593999 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:21.594823 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:21.595992 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:21.596367 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:21.596480 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:21.597058 (+ 578us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:49:21.597682 (+ 624us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:49:21.597807 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:21.597993 (+ 186us) server_negotiation.cc:299] Negotiation successful
0619 14:49:21.598079 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328}
I20260619 14:49:21.607136 13577 raft_consensus.cc:1261] T 43cfa821fb584abeb2b6581aaa247693 P 54040795b66642fcb61df15d9cf09111 [term 1 FOLLOWER]: Refusing update from remote peer 26305e9bb02641a4ace1ddf74f544f00: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:49:21.607136 13310 raft_consensus.cc:1261] T 43cfa821fb584abeb2b6581aaa247693 P 6e86100f05cb49eea64fb896fc8d484d [term 1 FOLLOWER]: Refusing update from remote peer 26305e9bb02641a4ace1ddf74f544f00: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:49:21.608728 13664 consensus_queue.cc:1035] T 43cfa821fb584abeb2b6581aaa247693 P 26305e9bb02641a4ace1ddf74f544f00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e86100f05cb49eea64fb896fc8d484d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:49:21.609266 13664 consensus_queue.cc:1035] T 43cfa821fb584abeb2b6581aaa247693 P 26305e9bb02641a4ace1ddf74f544f00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54040795b66642fcb61df15d9cf09111" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36461 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:49:21.623548 13673 mvcc.cc:205] Tried to move back new op lower bound from 7298582780331089920 to 7298582780148830208. Current Snapshot: MvccSnapshot[applied={T|T < 7298582780331089920}]
I20260619 14:49:21.646597 13155 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44760:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:49:21.646818 13155 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44760:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:49:21.650030 13155 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cb5d50f6090d4e8a8c49946022365fcb: Sending DeleteTablet for 3 replicas of tablet 43cfa821fb584abeb2b6581aaa247693
I20260619 14:49:21.651203 13556 tablet_service.cc:1514] Processing DeleteTablet for tablet 43cfa821fb584abeb2b6581aaa247693 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:49:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54362
I20260619 14:49:21.651631 13290 tablet_service.cc:1514] Processing DeleteTablet for tablet 43cfa821fb584abeb2b6581aaa247693 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:49:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42956
I20260619 14:49:21.652227 13424 tablet_service.cc:1514] Processing DeleteTablet for tablet 43cfa821fb584abeb2b6581aaa247693 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:49:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55076
I20260619 14:49:21.653249 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 13196
I20260619 14:49:21.657173 13684 tablet_replica.cc:330] T 43cfa821fb584abeb2b6581aaa247693 P 26305e9bb02641a4ace1ddf74f544f00: stopping tablet replica
I20260619 14:49:21.657567 13684 raft_consensus.cc:2229] T 43cfa821fb584abeb2b6581aaa247693 P 26305e9bb02641a4ace1ddf74f544f00 [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:49:21.658054 13684 raft_consensus.cc:2258] T 43cfa821fb584abeb2b6581aaa247693 P 26305e9bb02641a4ace1ddf74f544f00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:49:21.661026 13684 ts_tablet_manager.cc:1905] T 43cfa821fb584abeb2b6581aaa247693 P 26305e9bb02641a4ace1ddf74f544f00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:49:21.664338 13684 ts_tablet_manager.cc:1918] T 43cfa821fb584abeb2b6581aaa247693 P 26305e9bb02641a4ace1ddf74f544f00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:49:21.664486 13684 log.cc:1196] T 43cfa821fb584abeb2b6581aaa247693 P 26305e9bb02641a4ace1ddf74f544f00: Deleting WAL directory at /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ts-1/wal/wals/43cfa821fb584abeb2b6581aaa247693
I20260619 14:49:21.664933 13684 ts_tablet_manager.cc:1939] T 43cfa821fb584abeb2b6581aaa247693 P 26305e9bb02641a4ace1ddf74f544f00: Deleting consensus metadata
W20260619 14:49:21.666170 13143 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34993 (error 108)
I20260619 14:49:21.667699 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 13358
I20260619 14:49:21.667776 13655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:21.667417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34993 (local address 127.0.0.1:42966)
0619 14:49:21.667577 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:21.667669 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260619 14:49:21.667980 13143 catalog_manager.cc:4584] TS 6e86100f05cb49eea64fb896fc8d484d (127.0.98.1:34993): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 43cfa821fb584abeb2b6581aaa247693: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34993: connect: Connection refused (error 111)
I20260619 14:49:21.668980 13682 tablet_replica.cc:330] T 43cfa821fb584abeb2b6581aaa247693 P 54040795b66642fcb61df15d9cf09111: stopping tablet replica
I20260619 14:49:21.669394 13682 raft_consensus.cc:2229] T 43cfa821fb584abeb2b6581aaa247693 P 54040795b66642fcb61df15d9cf09111 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:49:21.669809 13682 raft_consensus.cc:2258] T 43cfa821fb584abeb2b6581aaa247693 P 54040795b66642fcb61df15d9cf09111 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:49:21.678931 13682 ts_tablet_manager.cc:1905] T 43cfa821fb584abeb2b6581aaa247693 P 54040795b66642fcb61df15d9cf09111: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:49:21.681983 13682 ts_tablet_manager.cc:1918] T 43cfa821fb584abeb2b6581aaa247693 P 54040795b66642fcb61df15d9cf09111: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:49:21.682139 13682 log.cc:1196] T 43cfa821fb584abeb2b6581aaa247693 P 54040795b66642fcb61df15d9cf09111: Deleting WAL directory at /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781880383132139-392-0/minicluster-data/ts-2/wal/wals/43cfa821fb584abeb2b6581aaa247693
I20260619 14:49:21.682456 13682 ts_tablet_manager.cc:1939] T 43cfa821fb584abeb2b6581aaa247693 P 54040795b66642fcb61df15d9cf09111: Deleting consensus metadata
I20260619 14:49:21.684049 13142 catalog_manager.cc:4841] TS 54040795b66642fcb61df15d9cf09111 (127.0.98.3:36461): tablet 43cfa821fb584abeb2b6581aaa247693 (table test-table [id=f1679e7f20514cc482e8ade9f9521876]) successfully deleted
I20260619 14:49:21.701081 13655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:21.700708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40879 (local address 127.0.0.1:55098)
0619 14:49:21.700918 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:21.700975 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:40879: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
I20260619 14:49:21.707238 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 13492
I20260619 14:49:21.717873 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 13125
I20260619 14:49:21.727478 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260619 14:49:22.074178 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260619 14:49:22.074319 392 mini_ranger.cc:66] Stopping Ranger...
I20260619 14:49:22.421047 392 mini_ranger.cc:68] Stopped Ranger
2026-06-19 14:49:22.421 UTC [12192] LOG: received smart shutdown request
2026-06-19 14:49:22.425 UTC [12192] LOG: background worker "logical replication launcher" (PID 12201) exited with exit code 1
2026-06-19 14:49:22.425 UTC [12196] LOG: shutting down
2026-06-19 14:49:22.649 UTC [12192] LOG: database system is shut down
2026-06-19T14:49:22Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (55920 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880383132139-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:49:22 dist-test-slave-2sb1 krb5kdc[13694](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:49:22 dist-test-slave-2sb1 krb5kdc[13694](info): set up 2 sockets
Jun 19 14:49:22 dist-test-slave-2sb1 krb5kdc[13694](info): commencing operation
krb5kdc: starting...
W20260619 14:49:23.865482 392 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880383132139-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880383132139-392-0/krb5kdc/test-user.keytab.
Jun 19 14:49:23 dist-test-slave-2sb1 krb5kdc[13694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880563, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:49:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:49:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 14:49:24.009578 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880383132139-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880383132139-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880383132139-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42521
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42521,127.0.98.61:46073,127.0.98.60:42059
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:49:24.131971 13707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:49:24.132359 13707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:49:24.132452 13707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:49:24.136852 13707 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:49:24.136978 13707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:49:24.137015 13707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:49:24.137058 13707 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:49:24.137090 13707 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:49:24.142879 13707 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880383132139-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880383132139-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42521,127.0.98.61:46073,127.0.98.60:42059
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42521
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13707
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880383132139-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:49:24.144109 13707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:49:24.145248 13707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:49:24.154836 13713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:49:24.155059 13707 server_base.cc:1034] running on GCE node
W20260619 14:49:24.154824 13712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:49:24.154855 13715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:49:24.155751 13707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:49:24.157071 13707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:49:24.158253 13707 hybrid_clock.cc:648] HybridClock initialized: now 1781880564158220 us; error 50 us; skew 500 ppm
Jun 19 14:49:24 dist-test-slave-2sb1 krb5kdc[13694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880564, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:49:24.161803 13707 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 14:49:24.162990 13707 webserver.cc:460] Webserver started at http://127.0.98.62:33051/ using document root <none> and password file <none>
I20260619 14:49:24.163780 13707 fs_manager.cc:359] Metadata directory not provided
I20260619 14:49:24.163900 13707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:49:24.164171 13707 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:49:24.166676 13707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880383132139-392-0/minicluster-data/master-0/data/instance:
uuid: "2cc2c60998dc48c290173f9ccd8dda0d"
format_stamp: "Formatted at 2026-06-19 14:49:24 on dist-test-slave-2sb1"
server_key: "42777c0fec4284738ccd89ae4bcdc90a"
server_key_iv: "51bf70b4880aff6e7f635089c14175ab"
server_key_version: "encryptionkey@0"
I20260619 14:49:24.167375 13707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880383132139-392-0/minicluster-data/master-0/wal/instance:
uuid: "2cc2c60998dc48c290173f9ccd8dda0d"
format_stamp: "Formatted at 2026-06-19 14:49:24 on dist-test-slave-2sb1"
server_key: "42777c0fec4284738ccd89ae4bcdc90a"
server_key_iv: "51bf70b4880aff6e7f635089c14175ab"
server_key_version: "encryptionkey@0"
I20260619 14:49:24.171996 13707 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260619 14:49:24.175395 13722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:24.177608 13707 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 14:49:24.178006 13707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880383132139-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880383132139-392-0/minicluster-data/master-0/wal
uuid: "2cc2c60998dc48c290173f9ccd8dda0d"
format_stamp: "Formatted at 2026-06-19 14:49:24 on dist-test-slave-2sb1"
server_key: "42777c0fec4284738ccd89ae4bcdc90a"
server_key_iv: "51bf70b4880aff6e7f635089c14175ab"
server_key_version: "encryptionkey@0"
I20260619 14:49:24.178176 13707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880383132139-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880383132139-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880383132139-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:49:24.194427 13707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:49:24.198439 13707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:49:24.198684 13707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:49:24.206730 13707 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42521
I20260619 14:49:24.206754 13774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42521 every 8 connection(s)
I20260619 14:49:24.207758 13707 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:49:24.208266 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 13707
I20260619 14:49:24.208411 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880383132139-392-0/minicluster-data/master-0/wal/instance
I20260619 14:49:24.208715 392 external_mini_cluster.cc:1428] Setting key 685d5625c668ae59a6e7a38461e7e320
I20260619 14:49:24.211217 13775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:49:24.215804 13775 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42521 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Jun 19 14:49:24 dist-test-slave-2sb1 krb5kdc[13694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880564, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:49:24.236887 13779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:24.218761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43918 (local address 127.0.98.62:42521)
0619 14:49:24.219278 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:24.219298 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:24.219547 (+ 249us) server_negotiation.cc:407] Connection header received
0619 14:49:24.220959 (+ 1412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:24.220969 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:24.221262 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:24.221465 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:49:24.222958 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:24.224581 (+ 1623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:24.226489 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:24.227038 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:24.227204 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:24.230538 (+ 3334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:49:24.230546 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:49:24.230563 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:49:24.230594 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:49:24.233064 (+ 2470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:24.233728 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:24.233735 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:24.233737 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:24.233908 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:24.234282 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:24.234285 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:24.234286 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:24.234863 (+ 577us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:49:24.234984 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:24.235591 (+ 607us) server_negotiation.cc:299] Negotiation successful
0619 14:49:24.235796 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"spinlock_wait_cycles":13184,"thread_start_us":201,"threads_started":1}
I20260619 14:49:24.236884 13778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:24.218776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42521 (local address 127.0.0.1:43918)
0619 14:49:24.219332 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:24.219389 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 14:49:24.220487 (+ 1098us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:49:24.221526 (+ 1039us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:49:24.221535 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:49:24.222263 (+ 728us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:49:24.222705 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:24.222730 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:24.224830 (+ 2100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:24.224837 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:24.226226 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:24.226328 (+ 102us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:24.227255 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:24.227263 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:24.227435 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:24.228248 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:49:24.228271 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:49:24.230330 (+ 2059us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:49:24.233253 (+ 2923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:49:24.233262 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:49:24.233273 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:49:24.233583 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:49:24.234023 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:49:24.234026 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:49:24.234027 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:49:24.234151 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:49:24.234996 (+ 845us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:49:24.235002 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:49:24.235164 (+ 162us) client_negotiation.cc:769] Sending connection context
0619 14:49:24.235407 (+ 243us) client_negotiation.cc:240] Negotiation successful
0619 14:49:24.235979 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":164,"threads_started":1}
I20260619 14:49:24.239833 13775 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46073 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:49:24.240854 13778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:24.240428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46073 (local address 127.0.0.1:55276)
0619 14:49:24.240588 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:24.240760 (+ 172us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880383132139-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880383132139-392-0/krb5kdc/kudu_127.0.98.61.keytab.
W20260619 14:49:24.241161 13725 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46073: connect: Connection refused (error 111)
I20260619 14:49:24.242313 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880383132139-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880383132139-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880383132139-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880383132139-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:46073
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880383132139-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42521,127.0.98.61:46073,127.0.98.60:42059
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:49:24.242650 13778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:24.242128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46073 (local address 127.0.0.1:55278)
0619 14:49:24.242254 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:24.242559 (+ 305us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260619 14:49:24.242820 13775 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:46073: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46073: connect: Connection refused (error 111)
I20260619 14:49:24.292168 13775 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46073 } attempt: 1
I20260619 14:49:24.293761 13778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:24.293356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46073 (local address 127.0.0.1:55294)
0619 14:49:24.293587 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:24.293647 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140}
I20260619 14:49:24.295415 13778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:24.294842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46073 (local address 127.0.0.1:55308)
0619 14:49:24.295021 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:24.295121 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260619 14:49:24.295625 13775 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:46073: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46073: connect: Connection refused (error 111)
W20260619 14:49:24.363545 13782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:49:24.363914 13782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
I20260619 14:49:24.363818 13775 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46073 } attempt: 2
W20260619 14:49:24.364013 13782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260619 14:49:24.364880 13778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:24.364509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46073 (local address 127.0.0.1:55312)
0619 14:49:24.364713 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:24.364784 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260619 14:49:24.365655 13778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:24.365418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46073 (local address 127.0.0.1:55316)
0619 14:49:24.365531 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:24.365582 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260619 14:49:24.365798 13775 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:46073: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46073: connect: Connection refused (error 111)
W20260619 14:49:24.368628 13782 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:49:24.368795 13782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:49:24.368840 13782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:49:24.368889 13782 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:49:24.368927 13782 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:49:24.374969 13782 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880383132139-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880383132139-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42521,127.0.98.61:46073,127.0.98.60:42059
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880383132139-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:46073
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880383132139-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13782
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880383132139-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:49:24.376195 13782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:49:24.377326 13782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:49:24.386807 13787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:49:24.386905 13788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:49:24.386880 13790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:49:24.387354 13782 server_base.cc:1034] running on GCE node
I20260619 14:49:24.388101 13782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:49:24.389950 13782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:49:24.391153 13782 hybrid_clock.cc:648] HybridClock initialized: now 1781880564391136 us; error 45 us; skew 500 ppm
Jun 19 14:49:24 dist-test-slave-2sb1 krb5kdc[13694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880564, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:49:24.394842 13782 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260619 14:49:24.396080 13782 webserver.cc:460] Webserver started at http://127.0.98.61:42663/ using document root <none> and password file <none>
I20260619 14:49:24.396790 13782 fs_manager.cc:359] Metadata directory not provided
I20260619 14:49:24.396864 13782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:49:24.397105 13782 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:49:24.399340 13782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880383132139-392-0/minicluster-data/master-1/data/instance:
uuid: "faace69b08ca47c0af6e754be5aed037"
format_stamp: "Formatted at 2026-06-19 14:49:24 on dist-test-slave-2sb1"
server_key: "d9545f20a0185262dee11521f9c3e6cf"
server_key_iv: "b1ebe92514768f3f481db104f74f6e98"
server_key_version: "encryptionkey@0"
I20260619 14:49:24.400041 13782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880383132139-392-0/minicluster-data/master-1/wal/instance:
uuid: "faace69b08ca47c0af6e754be5aed037"
format_stamp: "Formatted at 2026-06-19 14:49:24 on dist-test-slave-2sb1"
server_key: "d9545f20a0185262dee11521f9c3e6cf"
server_key_iv: "b1ebe92514768f3f481db104f74f6e98"
server_key_version: "encryptionkey@0"
I20260619 14:49:24.404245 13782 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 14:49:24.407375 13797 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:24.409340 13782 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:49:24.409495 13782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880383132139-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880383132139-392-0/minicluster-data/master-1/wal
uuid: "faace69b08ca47c0af6e754be5aed037"
format_stamp: "Formatted at 2026-06-19 14:49:24 on dist-test-slave-2sb1"
server_key: "d9545f20a0185262dee11521f9c3e6cf"
server_key_iv: "b1ebe92514768f3f481db104f74f6e98"
server_key_version: "encryptionkey@0"
I20260619 14:49:24.409624 13782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880383132139-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880383132139-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880383132139-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:49:24.424013 13782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:49:24.427913 13782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:49:24.428153 13782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:49:24.437997 13782 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:46073
I20260619 14:49:24.438021 13849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:46073 every 8 connection(s)
I20260619 14:49:24.439105 13782 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880383132139-392-0/minicluster-data/master-1/data/info.pb
I20260619 14:49:24.440510 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 13782
I20260619 14:49:24.440685 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880383132139-392-0/minicluster-data/master-1/wal/instance
I20260619 14:49:24.441004 392 external_mini_cluster.cc:1428] Setting key f37e750a8a327848f4cb3f0bd3e9cce5
I20260619 14:49:24.442852 13850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:49:24.447069 13850 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42521 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260619 14:49:24.456956 13775 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46073 } attempt: 3
Jun 19 14:49:24 dist-test-slave-2sb1 krb5kdc[13694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880564, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:49:24.467015 13779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:24.449705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43920 (local address 127.0.98.62:42521)
0619 14:49:24.449949 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:24.449953 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:24.450954 (+ 1001us) server_negotiation.cc:407] Connection header received
0619 14:49:24.452164 (+ 1210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:24.452167 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:24.452428 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:24.452593 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:49:24.454710 (+ 2117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:24.455636 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:24.457587 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:24.457968 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:24.458072 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:24.460848 (+ 2776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:49:24.460852 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:49:24.460854 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:49:24.460880 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:49:24.464315 (+ 3435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:24.465060 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:24.465064 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:24.465065 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:24.465209 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:24.465592 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:24.465595 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:24.465596 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:24.465805 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:49:24.465892 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:24.466587 (+ 695us) server_negotiation.cc:299] Negotiation successful
0619 14:49:24.466731 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
Jun 19 14:49:24 dist-test-slave-2sb1 krb5kdc[13694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880564, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 14:49:24.468017 13853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:24.449985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42521 (local address 127.0.0.1:43920)
0619 14:49:24.450679 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:24.450727 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 14:49:24.451931 (+ 1204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:49:24.452910 (+ 979us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:49:24.452930 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:49:24.453634 (+ 704us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:49:24.454476 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:24.454503 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:24.455868 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:24.455872 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:24.457300 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:24.457388 (+ 88us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:24.458250 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:24.458253 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:24.458404 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:24.459114 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:49:24.459132 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:49:24.460630 (+ 1498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:49:24.464468 (+ 3838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:49:24.464476 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:49:24.464497 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:49:24.464932 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:49:24.465317 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:49:24.465319 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:49:24.465320 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:49:24.465476 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:49:24.465998 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:49:24.466005 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:49:24.466283 (+ 278us) client_negotiation.cc:769] Sending connection context
0619 14:49:24.466507 (+ 224us) client_negotiation.cc:240] Negotiation successful
0619 14:49:24.466985 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":146,"threads_started":1}
I20260619 14:49:24.469969 13850 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46073 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:49:24.472468 13778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:24.457527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46073 (local address 127.0.0.1:55330)
0619 14:49:24.457781 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:24.457799 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 14:49:24.457897 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:49:24.458826 (+ 929us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:49:24.458831 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:49:24.459060 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:49:24.459218 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:24.459229 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:24.461345 (+ 2116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:24.461352 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:24.462124 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:24.462136 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:24.463172 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:24.463179 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:24.463689 (+ 510us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:24.464126 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:49:24.464146 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:49:24.467583 (+ 3437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:49:24.470080 (+ 2497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:49:24.470084 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:49:24.470086 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:49:24.470416 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:49:24.470868 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:49:24.470875 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:49:24.470878 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:49:24.471050 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:49:24.471814 (+ 764us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:49:24.471817 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:49:24.472013 (+ 196us) client_negotiation.cc:769] Sending connection context
0619 14:49:24.472087 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 14:49:24.472216 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149}
I20260619 14:49:24.473423 13855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:24.457662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55330 (local address 127.0.98.61:46073)
0619 14:49:24.458208 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:24.458220 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:24.458246 (+ 26us) server_negotiation.cc:407] Connection header received
0619 14:49:24.458301 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:24.458309 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:24.458577 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:24.459498 (+ 921us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:49:24.459812 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:24.461190 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:24.462310 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:24.462988 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:24.463140 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:24.467739 (+ 4599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:49:24.467756 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:49:24.467767 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:49:24.467790 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:49:24.469955 (+ 2165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:24.470628 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:24.470635 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:24.470637 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:24.470742 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:24.471198 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:24.471200 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:24.471201 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:24.471659 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:49:24.472819 (+ 1160us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:24.473054 (+ 235us) server_negotiation.cc:299] Negotiation successful
0619 14:49:24.473237 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":188,"threads_started":1}
I20260619 14:49:24.475924 13775 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42059 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:49:24.477054 13778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:24.476546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42059 (local address 127.0.0.1:50862)
0619 14:49:24.476880 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:24.476936 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880383132139-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880383132139-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Jun 19 14:49:24 dist-test-slave-2sb1 krb5kdc[13694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880564, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 14:49:24.478161 13778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:24.477851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42059 (local address 127.0.0.1:50874)
0619 14:49:24.478011 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:24.478069 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260619 14:49:24.478617 13775 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42059: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42059: connect: Connection refused (error 111)
I20260619 14:49:24.478722 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880383132139-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880383132139-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880383132139-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880383132139-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:42059
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880383132139-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42521,127.0.98.61:46073,127.0.98.60:42059
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:49:24.481760 13853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:24.470586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46073 (local address 127.0.0.1:55346)
0619 14:49:24.470728 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:24.470745 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 14:49:24.470825 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:49:24.471381 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:49:24.471384 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:49:24.471559 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:49:24.471674 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:24.471680 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:24.473037 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:24.473041 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:24.473740 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:24.473750 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:24.475078 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:24.475081 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:24.475154 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:24.475789 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:49:24.475803 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:49:24.477721 (+ 1918us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:49:24.479902 (+ 2181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:49:24.479906 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:49:24.479908 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:49:24.480144 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:49:24.480464 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:49:24.480470 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:49:24.480473 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:49:24.480588 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:49:24.481250 (+ 662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:49:24.481253 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:49:24.481376 (+ 123us) client_negotiation.cc:769] Sending connection context
0619 14:49:24.481468 (+ 92us) client_negotiation.cc:240] Negotiation successful
0619 14:49:24.481629 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260619 14:49:24.482173 13856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:24.470768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55346 (local address 127.0.98.61:46073)
0619 14:49:24.471179 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:24.471188 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:24.471200 (+ 12us) server_negotiation.cc:407] Connection header received
0619 14:49:24.471237 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:24.471239 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:24.471274 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:24.471373 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:49:24.471782 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:24.472919 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:24.473870 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:24.474940 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:24.475069 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:24.477898 (+ 2829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:49:24.477903 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:49:24.477906 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:49:24.477934 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:49:24.479738 (+ 1804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:24.480253 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:24.480256 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:24.480258 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:24.480341 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:24.480862 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:24.480866 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:24.480867 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:24.481134 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:49:24.481233 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:24.481578 (+ 345us) server_negotiation.cc:299] Negotiation successful
0619 14:49:24.481841 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":9,"server-negotiator.queue_time_us":305,"thread_start_us":126,"threads_started":1}
I20260619 14:49:24.483366 13850 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42059 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:49:24.484334 13853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:24.484028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42059 (local address 127.0.0.1:50886)
0619 14:49:24.484172 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:24.484270 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260619 14:49:24.484676 13801 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42059: connect: Connection refused (error 111)
I20260619 14:49:24.485683 13853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:24.485475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42059 (local address 127.0.0.1:50890)
0619 14:49:24.485584 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:24.485629 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260619 14:49:24.485862 13850 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42059: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42059: connect: Connection refused (error 111)
I20260619 14:49:24.509814 13775 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42059 } attempt: 1
I20260619 14:49:24.510890 13778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:24.510475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42059 (local address 127.0.0.1:50894)
0619 14:49:24.510743 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:24.510809 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260619 14:49:24.511924 13778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:24.511530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42059 (local address 127.0.0.1:50896)
0619 14:49:24.511790 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:24.511851 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260619 14:49:24.512084 13775 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42059: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42059: connect: Connection refused (error 111)
I20260619 14:49:24.535213 13850 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42059 } attempt: 1
I20260619 14:49:24.536463 13853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:24.536039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42059 (local address 127.0.0.1:50912)
0619 14:49:24.536216 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:24.536376 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260619 14:49:24.537438 13853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:24.537138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42059 (local address 127.0.0.1:50924)
0619 14:49:24.537304 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:24.537370 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260619 14:49:24.537609 13850 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42059: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42059: connect: Connection refused (error 111)
I20260619 14:49:24.587316 13775 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42059 } attempt: 2
I20260619 14:49:24.588407 13778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:24.588072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42059 (local address 127.0.0.1:50936)
0619 14:49:24.588262 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:24.588328 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260619 14:49:24.589541 13778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:24.589279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42059 (local address 127.0.0.1:50950)
0619 14:49:24.589410 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:24.589475 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260619 14:49:24.589846 13775 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42059: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42059: connect: Connection refused (error 111)
I20260619 14:49:24.605906 13850 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42059 } attempt: 2
I20260619 14:49:24.606978 13853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:24.606638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42059 (local address 127.0.0.1:50954)
0619 14:49:24.606787 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:24.606875 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260619 14:49:24.607875 13853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:24.607587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42059 (local address 127.0.0.1:50966)
0619 14:49:24.607717 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:24.607780 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260619 14:49:24.608027 13850 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42059: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42059: connect: Connection refused (error 111)
W20260619 14:49:24.609766 13857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:49:24.610095 13857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:49:24.610204 13857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:49:24.614742 13857 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:49:24.614841 13857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:49:24.614878 13857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:49:24.614933 13857 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:49:24.614969 13857 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:49:24.621148 13857 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880383132139-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880383132139-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42521,127.0.98.61:46073,127.0.98.60:42059
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880383132139-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:42059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880383132139-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13857
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880383132139-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:49:24.622397 13857 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:49:24.623495 13857 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:49:24.633235 13866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:49:24.633235 13863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:49:24.633235 13864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:49:24.633575 13857 server_base.cc:1034] running on GCE node
I20260619 14:49:24.634243 13857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:49:24.635491 13857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:49:24.636745 13857 hybrid_clock.cc:648] HybridClock initialized: now 1781880564636732 us; error 32 us; skew 500 ppm
Jun 19 14:49:24 dist-test-slave-2sb1 krb5kdc[13694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880564, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:49:24.640097 13857 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260619 14:49:24.641544 13857 webserver.cc:460] Webserver started at http://127.0.98.60:41421/ using document root <none> and password file <none>
I20260619 14:49:24.642304 13857 fs_manager.cc:359] Metadata directory not provided
I20260619 14:49:24.642416 13857 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:49:24.642666 13857 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:49:24.644986 13857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880383132139-392-0/minicluster-data/master-2/data/instance:
uuid: "b64661427bdd40929f9ca99b8b3951a1"
format_stamp: "Formatted at 2026-06-19 14:49:24 on dist-test-slave-2sb1"
server_key: "e5a282bb03d1450108148a8194a9e4e5"
server_key_iv: "8efc16cbaee9d3698ba49319bbee31d3"
server_key_version: "encryptionkey@0"
I20260619 14:49:24.645619 13857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880383132139-392-0/minicluster-data/master-2/wal/instance:
uuid: "b64661427bdd40929f9ca99b8b3951a1"
format_stamp: "Formatted at 2026-06-19 14:49:24 on dist-test-slave-2sb1"
server_key: "e5a282bb03d1450108148a8194a9e4e5"
server_key_iv: "8efc16cbaee9d3698ba49319bbee31d3"
server_key_version: "encryptionkey@0"
I20260619 14:49:24.650110 13857 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 14:49:24.653596 13873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:24.655936 13857 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260619 14:49:24.656157 13857 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880383132139-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880383132139-392-0/minicluster-data/master-2/wal
uuid: "b64661427bdd40929f9ca99b8b3951a1"
format_stamp: "Formatted at 2026-06-19 14:49:24 on dist-test-slave-2sb1"
server_key: "e5a282bb03d1450108148a8194a9e4e5"
server_key_iv: "8efc16cbaee9d3698ba49319bbee31d3"
server_key_version: "encryptionkey@0"
I20260619 14:49:24.656316 13857 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880383132139-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880383132139-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880383132139-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:49:24.669299 13857 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:49:24.673444 13857 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:49:24.673699 13857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:49:24.681968 13857 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:42059
I20260619 14:49:24.682152 13925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:42059 every 8 connection(s)
I20260619 14:49:24.683099 13857 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880383132139-392-0/minicluster-data/master-2/data/info.pb
I20260619 14:49:24.686369 13926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:49:24.687167 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 13857
I20260619 14:49:24.687352 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781880383132139-392-0/minicluster-data/master-2/wal/instance
I20260619 14:49:24.687638 392 external_mini_cluster.cc:1428] Setting key cf88a89129fb6f2b223ea0abbe83cecf
I20260619 14:49:24.689044 13775 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42059 } attempt: 3
I20260619 14:49:24.691705 13926 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42521 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 14:49:24 dist-test-slave-2sb1 krb5kdc[13694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880564, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 19 14:49:24 dist-test-slave-2sb1 krb5kdc[13694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880563, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 14:49:24.699209 13850 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42059 } attempt: 3
Jun 19 14:49:24 dist-test-slave-2sb1 krb5kdc[13694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880564, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:49:24.705839 13778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:24.689721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42059 (local address 127.0.0.1:50980)
0619 14:49:24.690245 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:24.690266 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0619 14:49:24.690407 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:49:24.692028 (+ 1621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:49:24.692031 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:49:24.692251 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:49:24.692381 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:24.692389 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:24.694480 (+ 2091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:24.694484 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:24.695361 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:24.695373 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:24.696224 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:24.696231 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:24.696320 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:24.697070 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:49:24.697085 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:49:24.699293 (+ 2208us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:49:24.703548 (+ 4255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:49:24.703552 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:49:24.703555 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:49:24.703810 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:49:24.704142 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:49:24.704148 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:49:24.704150 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:49:24.704244 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:49:24.705282 (+ 1038us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:49:24.705285 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:49:24.705382 (+ 97us) client_negotiation.cc:769] Sending connection context
0619 14:49:24.705502 (+ 120us) client_negotiation.cc:240] Negotiation successful
0619 14:49:24.705648 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342}
I20260619 14:49:24.706408 13779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:24.693708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43922 (local address 127.0.98.62:42521)
0619 14:49:24.693827 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:24.693831 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:24.694208 (+ 377us) server_negotiation.cc:407] Connection header received
0619 14:49:24.694567 (+ 359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:24.694570 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:24.694626 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:24.694731 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:49:24.695445 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:24.696179 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:24.697449 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:24.697887 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:24.698003 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:24.701631 (+ 3628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:49:24.701635 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:49:24.701638 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:49:24.701665 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:49:24.703388 (+ 1723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:24.704007 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:24.704010 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:24.704012 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:24.704093 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:24.704468 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:24.704471 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:24.704473 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:24.704850 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:49:24.704945 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:24.706128 (+ 1183us) server_negotiation.cc:299] Negotiation successful
0619 14:49:24.706242 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 14:49:24.706827 13930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:24.689768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50980 (local address 127.0.98.60:42059)
0619 14:49:24.690125 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:24.690135 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:24.690355 (+ 220us) server_negotiation.cc:407] Connection header received
0619 14:49:24.691019 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:24.691049 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:24.691543 (+ 494us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:24.692069 (+ 526us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:49:24.692596 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:24.694254 (+ 1658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:24.695478 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:24.696070 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:24.696596 (+ 526us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:24.699484 (+ 2888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:49:24.699491 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:49:24.699508 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:49:24.699539 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:49:24.703417 (+ 3878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:24.703938 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:24.703943 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:24.703948 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:24.704035 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:24.704375 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:24.704378 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:24.704381 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:24.705191 (+ 810us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:49:24.705288 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:24.705583 (+ 295us) server_negotiation.cc:299] Negotiation successful
0619 14:49:24.705729 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":346,"server-negotiator.queue_time_us":310,"spinlock_wait_cycles":45952,"thread_start_us":162,"threads_started":1}
I20260619 14:49:24.706892 13929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:24.689617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50974 (local address 127.0.98.60:42059)
0619 14:49:24.690113 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:24.690135 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:24.690179 (+ 44us) server_negotiation.cc:407] Connection header received
0619 14:49:24.691019 (+ 840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:24.691049 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:24.691543 (+ 494us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:24.692069 (+ 526us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:49:24.693361 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:24.694255 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:24.695541 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:24.696062 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:24.696251 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:24.700242 (+ 3991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:49:24.700248 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:49:24.700254 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:49:24.700285 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:49:24.702892 (+ 2607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:24.703540 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:24.703546 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:24.703548 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:24.703718 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:24.704214 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:24.704217 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:24.704218 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:24.704952 (+ 734us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:49:24.705167 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:24.706559 (+ 1392us) server_negotiation.cc:299] Negotiation successful
0619 14:49:24.706679 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1160,"server-negotiator.queue_time_us":332,"spinlock_wait_cycles":20480,"thread_start_us":146,"threads_started":1}
Jun 19 14:49:24 dist-test-slave-2sb1 krb5kdc[13694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880564, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 14:49:24.707918 13931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:24.693621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42521 (local address 127.0.0.1:43922)
0619 14:49:24.694082 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:24.694136 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0619 14:49:24.694388 (+ 252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:49:24.694818 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:49:24.694824 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:49:24.695195 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:49:24.695323 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:24.695330 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:24.696293 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:24.696296 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:24.697306 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:24.697317 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:24.698084 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:24.698089 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:24.698208 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:24.698955 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:49:24.698982 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:49:24.701460 (+ 2478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:49:24.703526 (+ 2066us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:49:24.703535 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:49:24.703547 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:49:24.703898 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:49:24.704197 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:49:24.704200 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:49:24.704202 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:49:24.704356 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:49:24.704997 (+ 641us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:49:24.705004 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:49:24.705162 (+ 158us) client_negotiation.cc:769] Sending connection context
0619 14:49:24.705342 (+ 180us) client_negotiation.cc:240] Negotiation successful
0619 14:49:24.705679 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"mutex_wait_us":1,"spinlock_wait_cycles":3968,"thread_start_us":129,"threads_started":1}
I20260619 14:49:24.709107 13926 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46073 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:49:24.712046 13775 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cc2c60998dc48c290173f9ccd8dda0d: Bootstrap starting.
Jun 19 14:49:24 dist-test-slave-2sb1 krb5kdc[13694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880564, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 14:49:24.715704 13775 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2cc2c60998dc48c290173f9ccd8dda0d: Neither blocks nor log segments found. Creating new log.
I20260619 14:49:24.717077 13775 log.cc:824] T 00000000000000000000000000000000 P 2cc2c60998dc48c290173f9ccd8dda0d: Log is configured to *not* fsync() on all Append() calls
Jun 19 14:49:24 dist-test-slave-2sb1 krb5kdc[13694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880563, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:49:24.720088 13856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:24.709740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55350 (local address 127.0.98.61:46073)
0619 14:49:24.710046 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:24.710049 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:24.710439 (+ 390us) server_negotiation.cc:407] Connection header received
0619 14:49:24.710562 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:24.710565 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:24.710612 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:24.710706 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:49:24.711123 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:24.711681 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:24.712475 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:24.712888 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:24.712993 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:24.715406 (+ 2413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:49:24.715410 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:49:24.715413 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:49:24.715444 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:49:24.717725 (+ 2281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:24.718339 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:24.718343 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:24.718345 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:24.718440 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:24.718876 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:24.718880 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:24.718882 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:24.719254 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:49:24.719367 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:24.719747 (+ 380us) server_negotiation.cc:299] Negotiation successful
0619 14:49:24.719897 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220}
I20260619 14:49:24.720731 13931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:24.709621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46073 (local address 127.0.0.1:55350)
0619 14:49:24.710358 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:24.710369 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 14:49:24.710476 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:49:24.710720 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:49:24.710723 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:49:24.710906 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:49:24.711004 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:24.711009 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:24.711809 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:24.711813 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:24.712354 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:24.712361 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:24.712995 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:24.712999 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:24.713092 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:24.713573 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:49:24.713584 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:49:24.715149 (+ 1565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:49:24.717909 (+ 2760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:49:24.717913 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:49:24.717915 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:49:24.718236 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:49:24.718616 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:49:24.718621 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:49:24.718625 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:49:24.718739 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:49:24.719419 (+ 680us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:49:24.719422 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:49:24.719540 (+ 118us) client_negotiation.cc:769] Sending connection context
0619 14:49:24.720337 (+ 797us) client_negotiation.cc:240] Negotiation successful
0619 14:49:24.720501 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":663}
I20260619 14:49:24.720935 13775 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cc2c60998dc48c290173f9ccd8dda0d: No bootstrap required, opened a new log
I20260619 14:49:24.721145 13853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:24.699842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42059 (local address 127.0.0.1:50994)
0619 14:49:24.700091 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:24.700109 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 14:49:24.700203 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:49:24.701313 (+ 1110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:49:24.701316 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:49:24.701531 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:49:24.701659 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:24.701667 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:24.702966 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:24.702970 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:24.703832 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:24.703845 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:24.704513 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:24.704520 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:24.704631 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:24.705843 (+ 1212us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:49:24.705858 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:49:24.707541 (+ 1683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:49:24.719147 (+ 11606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:49:24.719152 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:49:24.719162 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:49:24.719458 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:49:24.719782 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:49:24.719786 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:49:24.719787 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:49:24.719873 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:49:24.720614 (+ 741us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:49:24.720617 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:49:24.720763 (+ 146us) client_negotiation.cc:769] Sending connection context
0619 14:49:24.720855 (+ 92us) client_negotiation.cc:240] Negotiation successful
0619 14:49:24.720995 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142}
I20260619 14:49:24.721586 13926 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42059 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:49:24.722215 13932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:24.699957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50994 (local address 127.0.98.60:42059)
0619 14:49:24.701033 (+ 1076us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:24.701042 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:24.701055 (+ 13us) server_negotiation.cc:407] Connection header received
0619 14:49:24.701089 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:24.701092 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:24.701144 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:24.701769 (+ 625us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:49:24.701870 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:24.702818 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:24.703984 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:24.704356 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:24.704920 (+ 564us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:24.707685 (+ 2765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:49:24.707689 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:49:24.707692 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:49:24.707717 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:49:24.719033 (+ 11316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:24.719585 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:24.719588 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:24.719590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:24.719679 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:24.719982 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:24.719984 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:24.719987 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:24.720490 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:49:24.721778 (+ 1288us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:24.721874 (+ 96us) server_negotiation.cc:299] Negotiation successful
0619 14:49:24.722013 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1002,"thread_start_us":136,"threads_started":1}
I20260619 14:49:24.722364 13779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:24.709879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43924 (local address 127.0.98.62:42521)
0619 14:49:24.710009 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:24.710014 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:24.710197 (+ 183us) server_negotiation.cc:407] Connection header received
0619 14:49:24.710290 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:24.710293 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:24.710340 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:24.711285 (+ 945us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:49:24.711380 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:24.712103 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:24.713248 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:24.713914 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:24.714387 (+ 473us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:24.717934 (+ 3547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:49:24.717938 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:49:24.717941 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:49:24.717972 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:49:24.719774 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:24.720308 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:24.720311 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:24.720313 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:24.720402 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:24.720787 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:24.720790 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:24.720792 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:24.721031 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:49:24.721213 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:24.722064 (+ 851us) server_negotiation.cc:299] Negotiation successful
0619 14:49:24.722209 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260619 14:49:24.726042 13850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P faace69b08ca47c0af6e754be5aed037: Bootstrap starting.
Jun 19 14:49:24 dist-test-slave-2sb1 krb5kdc[13694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880564, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 14:49:24.728516 13775 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2cc2c60998dc48c290173f9ccd8dda0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cc2c60998dc48c290173f9ccd8dda0d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42521 } } peers { permanent_uuid: "faace69b08ca47c0af6e754be5aed037" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46073 } } peers { permanent_uuid: "b64661427bdd40929f9ca99b8b3951a1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42059 } }
I20260619 14:49:24.728840 13775 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2cc2c60998dc48c290173f9ccd8dda0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:49:24.728911 13775 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2cc2c60998dc48c290173f9ccd8dda0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cc2c60998dc48c290173f9ccd8dda0d, State: Initialized, Role: FOLLOWER
I20260619 14:49:24.729261 13850 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P faace69b08ca47c0af6e754be5aed037: Neither blocks nor log segments found. Creating new log.
I20260619 14:49:24.729514 13775 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2cc2c60998dc48c290173f9ccd8dda0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cc2c60998dc48c290173f9ccd8dda0d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42521 } } peers { permanent_uuid: "faace69b08ca47c0af6e754be5aed037" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46073 } } peers { permanent_uuid: "b64661427bdd40929f9ca99b8b3951a1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42059 } }
I20260619 14:49:24.730332 13850 log.cc:824] T 00000000000000000000000000000000 P faace69b08ca47c0af6e754be5aed037: Log is configured to *not* fsync() on all Append() calls
Jun 19 14:49:24 dist-test-slave-2sb1 krb5kdc[13694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880563, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 14:49:24.731195 13775 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2cc2c60998dc48c290173f9ccd8dda0d [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:49:24.732466 13931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:24.722115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42059 (local address 127.0.0.1:50998)
0619 14:49:24.722865 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:24.722876 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 14:49:24.722985 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:49:24.723374 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:49:24.723378 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:49:24.723569 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:49:24.723670 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:24.723681 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:24.724562 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:24.724568 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:24.725417 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:24.725428 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:24.726070 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:24.726074 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:24.726147 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:24.726999 (+ 852us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:49:24.727016 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:49:24.728810 (+ 1794us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:49:24.730791 (+ 1981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:49:24.730794 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:49:24.730797 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:49:24.731030 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:49:24.731334 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:49:24.731337 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:49:24.731338 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:49:24.731416 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:49:24.731863 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:49:24.731867 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:49:24.731948 (+ 81us) client_negotiation.cc:769] Sending connection context
0619 14:49:24.732060 (+ 112us) client_negotiation.cc:240] Negotiation successful
0619 14:49:24.732252 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":700}
I20260619 14:49:24.732918 13929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:24.722193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50998 (local address 127.0.98.60:42059)
0619 14:49:24.722321 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:24.722325 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:24.722948 (+ 623us) server_negotiation.cc:407] Connection header received
0619 14:49:24.723201 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:24.723204 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:24.723246 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:24.723325 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:49:24.723815 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:24.724452 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:24.725544 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:24.725864 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:24.725961 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:24.728982 (+ 3021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:49:24.728986 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:49:24.728989 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:49:24.729012 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:49:24.730656 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:24.731136 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:24.731139 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:24.731141 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:24.731236 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:24.731502 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:24.731504 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:24.731506 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:24.731729 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:49:24.731831 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:24.732597 (+ 766us) server_negotiation.cc:299] Negotiation successful
0619 14:49:24.732759 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 14:49:24.736399 13926 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b64661427bdd40929f9ca99b8b3951a1: Bootstrap starting.
I20260619 14:49:24.737421 13856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:24.723757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55366 (local address 127.0.98.61:46073)
0619 14:49:24.723879 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:24.723884 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:24.723895 (+ 11us) server_negotiation.cc:407] Connection header received
0619 14:49:24.723930 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:24.723932 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:24.723973 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:24.724053 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:49:24.724886 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:24.725593 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:24.727041 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:24.727368 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:24.727477 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:24.731912 (+ 4435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:49:24.731915 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:49:24.731919 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:49:24.731954 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:49:24.734747 (+ 2793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:24.735465 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:24.735469 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:24.735471 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:24.735572 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:24.736071 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:24.736074 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:24.736076 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:24.736351 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:49:24.736585 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:24.737020 (+ 435us) server_negotiation.cc:299] Negotiation successful
0619 14:49:24.737270 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260619 14:49:24.740262 13946 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2cc2c60998dc48c290173f9ccd8dda0d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260619 14:49:24.740360 13926 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b64661427bdd40929f9ca99b8b3951a1: Neither blocks nor log segments found. Creating new log.
W20260619 14:49:24.740391 13946 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:49:24.732933 13934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cc2c60998dc48c290173f9ccd8dda0d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cc2c60998dc48c290173f9ccd8dda0d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42521 } } peers { permanent_uuid: "faace69b08ca47c0af6e754be5aed037" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46073 } } peers { permanent_uuid: "b64661427bdd40929f9ca99b8b3951a1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42059 } } }
I20260619 14:49:24.740939 13934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cc2c60998dc48c290173f9ccd8dda0d [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:49:24.741689 13926 log.cc:824] T 00000000000000000000000000000000 P b64661427bdd40929f9ca99b8b3951a1: Log is configured to *not* fsync() on all Append() calls
I20260619 14:49:24.742678 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:49:24.737237 13707 master_runner.cc:186] Error getting master registration for 127.0.98.61:46073: OK, instance_id { permanent_uuid: "faace69b08ca47c0af6e754be5aed037" instance_seqno: 1781880564435127 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 14:49:24.742885 13707 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 14:49:24.745250 13850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P faace69b08ca47c0af6e754be5aed037: No bootstrap required, opened a new log
I20260619 14:49:24.750329 13850 raft_consensus.cc:348] T 00000000000000000000000000000000 P faace69b08ca47c0af6e754be5aed037 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cc2c60998dc48c290173f9ccd8dda0d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42521 } } peers { permanent_uuid: "faace69b08ca47c0af6e754be5aed037" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46073 } } peers { permanent_uuid: "b64661427bdd40929f9ca99b8b3951a1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42059 } }
I20260619 14:49:24.750612 13850 raft_consensus.cc:374] T 00000000000000000000000000000000 P faace69b08ca47c0af6e754be5aed037 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:49:24.750672 13850 raft_consensus.cc:729] T 00000000000000000000000000000000 P faace69b08ca47c0af6e754be5aed037 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: faace69b08ca47c0af6e754be5aed037, State: Initialized, Role: FOLLOWER
I20260619 14:49:24.750921 13926 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b64661427bdd40929f9ca99b8b3951a1: No bootstrap required, opened a new log
I20260619 14:49:24.751242 13850 consensus_queue.cc:260] T 00000000000000000000000000000000 P faace69b08ca47c0af6e754be5aed037 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cc2c60998dc48c290173f9ccd8dda0d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42521 } } peers { permanent_uuid: "faace69b08ca47c0af6e754be5aed037" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46073 } } peers { permanent_uuid: "b64661427bdd40929f9ca99b8b3951a1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42059 } }
I20260619 14:49:24.753002 13850 sys_catalog.cc:564] T 00000000000000000000000000000000 P faace69b08ca47c0af6e754be5aed037 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:49:24.755153 13779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:24.744912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43928 (local address 127.0.98.62:42521)
0619 14:49:24.745039 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:24.745044 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:24.745576 (+ 532us) server_negotiation.cc:407] Connection header received
0619 14:49:24.745823 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:24.745829 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:24.745890 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:24.745949 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:49:24.747027 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:24.747543 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:24.748407 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:24.748742 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:24.748846 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:24.750909 (+ 2063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:49:24.750913 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:49:24.750915 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:49:24.750948 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:49:24.752919 (+ 1971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:24.753580 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:24.753584 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:24.753586 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:24.753700 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:24.754017 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:24.754020 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:24.754022 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:24.754259 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:49:24.754399 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:24.754809 (+ 410us) server_negotiation.cc:299] Negotiation successful
0619 14:49:24.754954 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 14:49:24.756973 13956 sys_catalog.cc:455] T 00000000000000000000000000000000 P faace69b08ca47c0af6e754be5aed037 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cc2c60998dc48c290173f9ccd8dda0d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42521 } } peers { permanent_uuid: "faace69b08ca47c0af6e754be5aed037" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46073 } } peers { permanent_uuid: "b64661427bdd40929f9ca99b8b3951a1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42059 } } }
I20260619 14:49:24.757143 13956 sys_catalog.cc:458] T 00000000000000000000000000000000 P faace69b08ca47c0af6e754be5aed037 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:49:24.757932 13926 raft_consensus.cc:348] T 00000000000000000000000000000000 P b64661427bdd40929f9ca99b8b3951a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cc2c60998dc48c290173f9ccd8dda0d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42521 } } peers { permanent_uuid: "faace69b08ca47c0af6e754be5aed037" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46073 } } peers { permanent_uuid: "b64661427bdd40929f9ca99b8b3951a1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42059 } }
I20260619 14:49:24.758216 13926 raft_consensus.cc:374] T 00000000000000000000000000000000 P b64661427bdd40929f9ca99b8b3951a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:49:24.758265 13926 raft_consensus.cc:729] T 00000000000000000000000000000000 P b64661427bdd40929f9ca99b8b3951a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b64661427bdd40929f9ca99b8b3951a1, State: Initialized, Role: FOLLOWER
I20260619 14:49:24.758847 13926 consensus_queue.cc:260] T 00000000000000000000000000000000 P b64661427bdd40929f9ca99b8b3951a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cc2c60998dc48c290173f9ccd8dda0d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42521 } } peers { permanent_uuid: "faace69b08ca47c0af6e754be5aed037" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46073 } } peers { permanent_uuid: "b64661427bdd40929f9ca99b8b3951a1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42059 } }
I20260619 14:49:24.760715 13926 sys_catalog.cc:564] T 00000000000000000000000000000000 P b64661427bdd40929f9ca99b8b3951a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:49:24.766012 13856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:24.746117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55382 (local address 127.0.98.61:46073)
0619 14:49:24.746252 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:24.746257 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:24.756227 (+ 9970us) server_negotiation.cc:407] Connection header received
0619 14:49:24.756425 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:24.756429 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:24.756481 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:24.756533 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:49:24.757652 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:24.758373 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:24.759636 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:24.759978 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:24.760057 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:24.762120 (+ 2063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:49:24.762124 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:49:24.762127 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:49:24.762156 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:49:24.764050 (+ 1894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:24.764499 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:24.764503 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:24.764504 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:24.764590 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:24.764909 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:24.764913 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:24.764915 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:24.765223 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:49:24.765352 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:24.765726 (+ 374us) server_negotiation.cc:299] Negotiation successful
0619 14:49:24.765857 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 14:49:24.762290 13782 master_runner.cc:186] Error getting master registration for 127.0.98.60:42059: OK, instance_id { permanent_uuid: "b64661427bdd40929f9ca99b8b3951a1" instance_seqno: 1781880564679696 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 14:49:24.766341 13782 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 14:49:24.768890 13968 sys_catalog.cc:455] T 00000000000000000000000000000000 P b64661427bdd40929f9ca99b8b3951a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cc2c60998dc48c290173f9ccd8dda0d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42521 } } peers { permanent_uuid: "faace69b08ca47c0af6e754be5aed037" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46073 } } peers { permanent_uuid: "b64661427bdd40929f9ca99b8b3951a1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42059 } } }
I20260619 14:49:24.769788 13968 sys_catalog.cc:458] T 00000000000000000000000000000000 P b64661427bdd40929f9ca99b8b3951a1 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:49:24.771086 13857 master_runner.cc:263] No leader master found from master b64661427bdd40929f9ca99b8b3951a1
I20260619 14:49:24.771163 13857 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260619 14:49:24.772899 13967 catalog_manager.cc:1558] T 00000000000000000000000000000000 P faace69b08ca47c0af6e754be5aed037: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:49:24.773073 13967 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:49:24.777508 13968 raft_consensus.cc:482] T 00000000000000000000000000000000 P b64661427bdd40929f9ca99b8b3951a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:49:24.777688 13968 raft_consensus.cc:504] T 00000000000000000000000000000000 P b64661427bdd40929f9ca99b8b3951a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cc2c60998dc48c290173f9ccd8dda0d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42521 } } peers { permanent_uuid: "faace69b08ca47c0af6e754be5aed037" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46073 } } peers { permanent_uuid: "b64661427bdd40929f9ca99b8b3951a1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42059 } }
I20260619 14:49:24.778484 13929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:24.748798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51002 (local address 127.0.98.60:42059)
0619 14:49:24.748944 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:24.748948 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:24.766711 (+ 17763us) server_negotiation.cc:407] Connection header received
0619 14:49:24.767237 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:24.767242 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:24.767295 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:24.767346 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:49:24.768372 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:24.769205 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:24.771476 (+ 2271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:24.771831 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:24.771903 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:24.773958 (+ 2055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:49:24.773962 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:49:24.773965 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:49:24.773996 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:49:24.776078 (+ 2082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:24.776734 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:24.776739 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:24.776741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:24.776842 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:24.777178 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:24.777182 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:24.777184 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:24.777435 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:49:24.777576 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:24.778170 (+ 594us) server_negotiation.cc:299] Negotiation successful
0619 14:49:24.778314 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 14:49:24.779263 13749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b64661427bdd40929f9ca99b8b3951a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cc2c60998dc48c290173f9ccd8dda0d" is_pre_election: true
I20260619 14:49:24.779827 13749 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2cc2c60998dc48c290173f9ccd8dda0d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b64661427bdd40929f9ca99b8b3951a1 in term 0.
I20260619 14:49:24.780355 13876 leader_election.cc:302] T 00000000000000000000000000000000 P b64661427bdd40929f9ca99b8b3951a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cc2c60998dc48c290173f9ccd8dda0d, b64661427bdd40929f9ca99b8b3951a1; no voters:
W20260619 14:49:24.780525 13979 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b64661427bdd40929f9ca99b8b3951a1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:49:24.780601 13979 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:49:24.780977 13968 leader_election.cc:288] T 00000000000000000000000000000000 P b64661427bdd40929f9ca99b8b3951a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2cc2c60998dc48c290173f9ccd8dda0d (127.0.98.62:42521), faace69b08ca47c0af6e754be5aed037 (127.0.98.61:46073)
I20260619 14:49:24.781083 13968 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b64661427bdd40929f9ca99b8b3951a1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:49:24.781149 13968 raft_consensus.cc:482] T 00000000000000000000000000000000 P b64661427bdd40929f9ca99b8b3951a1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:49:24.781191 13968 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b64661427bdd40929f9ca99b8b3951a1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:49:24.781577 13824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b64661427bdd40929f9ca99b8b3951a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "faace69b08ca47c0af6e754be5aed037" is_pre_election: true
I20260619 14:49:24.781967 13824 raft_consensus.cc:2454] T 00000000000000000000000000000000 P faace69b08ca47c0af6e754be5aed037 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b64661427bdd40929f9ca99b8b3951a1 in term 0.
I20260619 14:49:24.782792 13968 raft_consensus.cc:504] T 00000000000000000000000000000000 P b64661427bdd40929f9ca99b8b3951a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cc2c60998dc48c290173f9ccd8dda0d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42521 } } peers { permanent_uuid: "faace69b08ca47c0af6e754be5aed037" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46073 } } peers { permanent_uuid: "b64661427bdd40929f9ca99b8b3951a1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42059 } }
I20260619 14:49:24.783737 13749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b64661427bdd40929f9ca99b8b3951a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2cc2c60998dc48c290173f9ccd8dda0d"
I20260619 14:49:24.783723 13824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b64661427bdd40929f9ca99b8b3951a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "faace69b08ca47c0af6e754be5aed037"
I20260619 14:49:24.783885 13824 raft_consensus.cc:3037] T 00000000000000000000000000000000 P faace69b08ca47c0af6e754be5aed037 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:49:24.783877 13749 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2cc2c60998dc48c290173f9ccd8dda0d [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:49:24.785400 13749 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2cc2c60998dc48c290173f9ccd8dda0d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b64661427bdd40929f9ca99b8b3951a1 in term 1.
I20260619 14:49:24.785490 13824 raft_consensus.cc:2454] T 00000000000000000000000000000000 P faace69b08ca47c0af6e754be5aed037 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b64661427bdd40929f9ca99b8b3951a1 in term 1.
I20260619 14:49:24.785890 13876 leader_election.cc:302] T 00000000000000000000000000000000 P b64661427bdd40929f9ca99b8b3951a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b64661427bdd40929f9ca99b8b3951a1, faace69b08ca47c0af6e754be5aed037; no voters:
I20260619 14:49:24.786114 13968 leader_election.cc:288] T 00000000000000000000000000000000 P b64661427bdd40929f9ca99b8b3951a1 [CANDIDATE]: Term 1 election: Requested vote from peers 2cc2c60998dc48c290173f9ccd8dda0d (127.0.98.62:42521), faace69b08ca47c0af6e754be5aed037 (127.0.98.61:46073)
I20260619 14:49:24.786260 13968 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b64661427bdd40929f9ca99b8b3951a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:49:24.786332 13968 raft_consensus.cc:686] T 00000000000000000000000000000000 P b64661427bdd40929f9ca99b8b3951a1 [term 1 LEADER]: Becoming Leader. State: Replica: b64661427bdd40929f9ca99b8b3951a1, State: Running, Role: LEADER
I20260619 14:49:24.786675 13968 consensus_queue.cc:237] T 00000000000000000000000000000000 P b64661427bdd40929f9ca99b8b3951a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cc2c60998dc48c290173f9ccd8dda0d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42521 } } peers { permanent_uuid: "faace69b08ca47c0af6e754be5aed037" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46073 } } peers { permanent_uuid: "b64661427bdd40929f9ca99b8b3951a1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42059 } }
I20260619 14:49:24.789155 13968 sys_catalog.cc:455] T 00000000000000000000000000000000 P b64661427bdd40929f9ca99b8b3951a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b64661427bdd40929f9ca99b8b3951a1. Latest consensus state: current_term: 1 leader_uuid: "b64661427bdd40929f9ca99b8b3951a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cc2c60998dc48c290173f9ccd8dda0d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42521 } } peers { permanent_uuid: "faace69b08ca47c0af6e754be5aed037" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46073 } } peers { permanent_uuid: "b64661427bdd40929f9ca99b8b3951a1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42059 } } }
I20260619 14:49:24.789309 13968 sys_catalog.cc:458] T 00000000000000000000000000000000 P b64661427bdd40929f9ca99b8b3951a1 [sys.catalog]: This master's current role is: LEADER
I20260619 14:49:24.797073 13984 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:49:24.798275 13984 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:49:24.805181 13824 raft_consensus.cc:1261] T 00000000000000000000000000000000 P faace69b08ca47c0af6e754be5aed037 [term 1 FOLLOWER]: Refusing update from remote peer b64661427bdd40929f9ca99b8b3951a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:49:24.806008 13982 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b64661427bdd40929f9ca99b8b3951a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "faace69b08ca47c0af6e754be5aed037" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:49:24.810613 13956 sys_catalog.cc:455] T 00000000000000000000000000000000 P faace69b08ca47c0af6e754be5aed037 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b64661427bdd40929f9ca99b8b3951a1. Latest consensus state: current_term: 1 leader_uuid: "b64661427bdd40929f9ca99b8b3951a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cc2c60998dc48c290173f9ccd8dda0d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42521 } } peers { permanent_uuid: "faace69b08ca47c0af6e754be5aed037" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46073 } } peers { permanent_uuid: "b64661427bdd40929f9ca99b8b3951a1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42059 } } }
I20260619 14:49:24.810801 13956 sys_catalog.cc:458] T 00000000000000000000000000000000 P faace69b08ca47c0af6e754be5aed037 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:49:24.812155 13749 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2cc2c60998dc48c290173f9ccd8dda0d [term 1 FOLLOWER]: Refusing update from remote peer b64661427bdd40929f9ca99b8b3951a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:49:24.813009 13982 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b64661427bdd40929f9ca99b8b3951a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2cc2c60998dc48c290173f9ccd8dda0d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:49:24.818759 13934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cc2c60998dc48c290173f9ccd8dda0d [sys.catalog]: SysCatalogTable state changed. Reason: New leader b64661427bdd40929f9ca99b8b3951a1. Latest consensus state: current_term: 1 leader_uuid: "b64661427bdd40929f9ca99b8b3951a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cc2c60998dc48c290173f9ccd8dda0d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42521 } } peers { permanent_uuid: "faace69b08ca47c0af6e754be5aed037" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46073 } } peers { permanent_uuid: "b64661427bdd40929f9ca99b8b3951a1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42059 } } }
I20260619 14:49:24.818950 13934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cc2c60998dc48c290173f9ccd8dda0d [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:49:24.826411 13968 sys_catalog.cc:455] T 00000000000000000000000000000000 P b64661427bdd40929f9ca99b8b3951a1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b64661427bdd40929f9ca99b8b3951a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cc2c60998dc48c290173f9ccd8dda0d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42521 } } peers { permanent_uuid: "faace69b08ca47c0af6e754be5aed037" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46073 } } peers { permanent_uuid: "b64661427bdd40929f9ca99b8b3951a1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42059 } } }
I20260619 14:49:24.826632 13968 sys_catalog.cc:458] T 00000000000000000000000000000000 P b64661427bdd40929f9ca99b8b3951a1 [sys.catalog]: This master's current role is: LEADER
I20260619 14:49:24.827154 13934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cc2c60998dc48c290173f9ccd8dda0d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b64661427bdd40929f9ca99b8b3951a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cc2c60998dc48c290173f9ccd8dda0d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42521 } } peers { permanent_uuid: "faace69b08ca47c0af6e754be5aed037" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46073 } } peers { permanent_uuid: "b64661427bdd40929f9ca99b8b3951a1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42059 } } }
I20260619 14:49:24.827298 13934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cc2c60998dc48c290173f9ccd8dda0d [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:49:24.827670 13985 mvcc.cc:205] Tried to move back new op lower bound from 7298582793432547328 to 7298582793369726976. Current Snapshot: MvccSnapshot[applied={T|T < 7298582793432547328}]
I20260619 14:49:24.830062 13988 mvcc.cc:205] Tried to move back new op lower bound from 7298582793432547328 to 7298582793369726976. Current Snapshot: MvccSnapshot[applied={T|T < 7298582793432547328}]
I20260619 14:49:24.830968 13982 sys_catalog.cc:455] T 00000000000000000000000000000000 P b64661427bdd40929f9ca99b8b3951a1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b64661427bdd40929f9ca99b8b3951a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cc2c60998dc48c290173f9ccd8dda0d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42521 } } peers { permanent_uuid: "faace69b08ca47c0af6e754be5aed037" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46073 } } peers { permanent_uuid: "b64661427bdd40929f9ca99b8b3951a1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42059 } } }
I20260619 14:49:24.831110 13982 sys_catalog.cc:458] T 00000000000000000000000000000000 P b64661427bdd40929f9ca99b8b3951a1 [sys.catalog]: This master's current role is: LEADER
I20260619 14:49:24.832456 13956 sys_catalog.cc:455] T 00000000000000000000000000000000 P faace69b08ca47c0af6e754be5aed037 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b64661427bdd40929f9ca99b8b3951a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2cc2c60998dc48c290173f9ccd8dda0d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42521 } } peers { permanent_uuid: "faace69b08ca47c0af6e754be5aed037" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46073 } } peers { permanent_uuid: "b64661427bdd40929f9ca99b8b3951a1" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42059 } } }
I20260619 14:49:24.832597 13956 sys_catalog.cc:458] T 00000000000000000000000000000000 P faace69b08ca47c0af6e754be5aed037 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:49:24.833534 13986 mvcc.cc:205] Tried to move back new op lower bound from 7298582793432547328 to 7298582793369726976. Current Snapshot: MvccSnapshot[applied={T|T < 7298582793432547328}]
I20260619 14:49:24.842837 13984 catalog_manager.cc:1347] Generated new cluster ID: c5a6601e5d83416195c6806946030760
I20260619 14:49:24.842947 13984 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:49:24.868058 13984 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:49:24.869925 13984 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:49:24.895875 13984 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b64661427bdd40929f9ca99b8b3951a1: Generated new TSK 0
I20260619 14:49:24.897167 13984 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:49:25.741910 13946 catalog_manager.cc:1259] Loaded cluster ID: c5a6601e5d83416195c6806946030760
I20260619 14:49:25.742017 13946 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2cc2c60998dc48c290173f9ccd8dda0d: loading cluster ID for follower catalog manager: success
I20260619 14:49:25.743947 13946 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2cc2c60998dc48c290173f9ccd8dda0d: acquiring CA information for follower catalog manager: success
I20260619 14:49:25.745239 13946 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2cc2c60998dc48c290173f9ccd8dda0d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 14:49:25.774544 13967 catalog_manager.cc:1259] Loaded cluster ID: c5a6601e5d83416195c6806946030760
I20260619 14:49:25.774652 13967 catalog_manager.cc:1552] T 00000000000000000000000000000000 P faace69b08ca47c0af6e754be5aed037: loading cluster ID for follower catalog manager: success
I20260619 14:49:25.776727 13967 catalog_manager.cc:1574] T 00000000000000000000000000000000 P faace69b08ca47c0af6e754be5aed037: acquiring CA information for follower catalog manager: success
I20260619 14:49:25.778127 13967 catalog_manager.cc:1602] T 00000000000000000000000000000000 P faace69b08ca47c0af6e754be5aed037: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 14:49:25.975944 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 13707
W20260619 14:49:25.982213 13876 connection.cc:535] client connection to 127.0.98.62:42521 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260619 14:49:25.982437 13876 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 14:49:25.982468 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 13782
I20260619 14:49:25.983959 13999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:25.983543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42521 (local address 127.0.0.1:43936)
0619 14:49:25.983777 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:25.983844 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":68,"threads_started":1}
W20260619 14:49:25.984155 13876 consensus_peers.cc:487] T 00000000000000000000000000000000 P b64661427bdd40929f9ca99b8b3951a1 -> Peer 2cc2c60998dc48c290173f9ccd8dda0d (127.0.98.62:42521): Couldn't send request to peer 2cc2c60998dc48c290173f9ccd8dda0d. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:42521: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260619 14:49:25.988914 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 13857
2026-06-19T14:49:25Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3214 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781880383132139-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:49:26 dist-test-slave-2sb1 krb5kdc[14002](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:49:26 dist-test-slave-2sb1 krb5kdc[14002](info): set up 2 sockets
Jun 19 14:49:26 dist-test-slave-2sb1 krb5kdc[14002](info): commencing operation
krb5kdc: starting...
W20260619 14:49:27.089491 392 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781880383132139-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781880383132139-392-0/krb5kdc/test-user.keytab.
Jun 19 14:49:27 dist-test-slave-2sb1 krb5kdc[14002](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880567, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:49:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:49:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 14:49:27.238099 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781880383132139-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781880383132139-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781880383132139-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36633
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36633
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:49:27.358898 14015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:49:27.359268 14015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:49:27.359357 14015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:49:27.363816 14015 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:49:27.363957 14015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:49:27.363996 14015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:49:27.364032 14015 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:49:27.364071 14015 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:49:27.369947 14015 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781880383132139-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781880383132139-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36633
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14015
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781880383132139-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:49:27.371140 14015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:49:27.372208 14015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:49:27.381613 14023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:49:27.381676 14020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:49:27.381841 14015 server_base.cc:1034] running on GCE node
W20260619 14:49:27.381842 14021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:49:27.382815 14015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:49:27.384168 14015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:49:27.385416 14015 hybrid_clock.cc:648] HybridClock initialized: now 1781880567385368 us; error 66 us; skew 500 ppm
Jun 19 14:49:27 dist-test-slave-2sb1 krb5kdc[14002](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880567, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:49:27.393996 14015 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 14:49:27.395219 14015 webserver.cc:460] Webserver started at http://127.0.98.62:42933/ using document root <none> and password file <none>
I20260619 14:49:27.396020 14015 fs_manager.cc:359] Metadata directory not provided
I20260619 14:49:27.396134 14015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:49:27.396394 14015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:49:27.398823 14015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781880383132139-392-0/minicluster-data/master-0/data/instance:
uuid: "46c4e03eadd741d4baed25ca16cd4119"
format_stamp: "Formatted at 2026-06-19 14:49:27 on dist-test-slave-2sb1"
server_key: "b63207bd46acb2b4075a8c73ac42c132"
server_key_iv: "2181b5ae78b25fa334de9ea118578353"
server_key_version: "encryptionkey@0"
I20260619 14:49:27.399495 14015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781880383132139-392-0/minicluster-data/master-0/wal/instance:
uuid: "46c4e03eadd741d4baed25ca16cd4119"
format_stamp: "Formatted at 2026-06-19 14:49:27 on dist-test-slave-2sb1"
server_key: "b63207bd46acb2b4075a8c73ac42c132"
server_key_iv: "2181b5ae78b25fa334de9ea118578353"
server_key_version: "encryptionkey@0"
I20260619 14:49:27.404727 14015 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 14:49:27.407840 14030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:27.409472 14015 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:49:27.409664 14015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781880383132139-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781880383132139-392-0/minicluster-data/master-0/wal
uuid: "46c4e03eadd741d4baed25ca16cd4119"
format_stamp: "Formatted at 2026-06-19 14:49:27 on dist-test-slave-2sb1"
server_key: "b63207bd46acb2b4075a8c73ac42c132"
server_key_iv: "2181b5ae78b25fa334de9ea118578353"
server_key_version: "encryptionkey@0"
I20260619 14:49:27.409970 14015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781880383132139-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781880383132139-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781880383132139-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:49:27.457365 14015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:49:27.461203 14015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:49:27.461455 14015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:49:27.469327 14015 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36633
I20260619 14:49:27.469362 14082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36633 every 8 connection(s)
I20260619 14:49:27.470351 14015 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:49:27.473675 14083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:49:27.477073 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 14015
I20260619 14:49:27.477224 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781880383132139-392-0/minicluster-data/master-0/wal/instance
I20260619 14:49:27.477555 392 external_mini_cluster.cc:1428] Setting key 9c182d976c86989e2d70a6598668eb18
I20260619 14:49:27.480592 14083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46c4e03eadd741d4baed25ca16cd4119: Bootstrap starting.
I20260619 14:49:27.483525 14083 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 46c4e03eadd741d4baed25ca16cd4119: Neither blocks nor log segments found. Creating new log.
I20260619 14:49:27.484431 14083 log.cc:824] T 00000000000000000000000000000000 P 46c4e03eadd741d4baed25ca16cd4119: Log is configured to *not* fsync() on all Append() calls
I20260619 14:49:27.487044 14083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46c4e03eadd741d4baed25ca16cd4119: No bootstrap required, opened a new log
Jun 19 14:49:27 dist-test-slave-2sb1 krb5kdc[14002](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880567, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:49:27.491689 14083 raft_consensus.cc:348] T 00000000000000000000000000000000 P 46c4e03eadd741d4baed25ca16cd4119 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46c4e03eadd741d4baed25ca16cd4119" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36633 } }
I20260619 14:49:27.492031 14083 raft_consensus.cc:374] T 00000000000000000000000000000000 P 46c4e03eadd741d4baed25ca16cd4119 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:49:27.492117 14083 raft_consensus.cc:729] T 00000000000000000000000000000000 P 46c4e03eadd741d4baed25ca16cd4119 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46c4e03eadd741d4baed25ca16cd4119, State: Initialized, Role: FOLLOWER
I20260619 14:49:27.492624 14083 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46c4e03eadd741d4baed25ca16cd4119 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46c4e03eadd741d4baed25ca16cd4119" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36633 } }
I20260619 14:49:27.492825 14083 raft_consensus.cc:388] T 00000000000000000000000000000000 P 46c4e03eadd741d4baed25ca16cd4119 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:49:27.492930 14083 raft_consensus.cc:482] T 00000000000000000000000000000000 P 46c4e03eadd741d4baed25ca16cd4119 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:49:27.493058 14083 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 46c4e03eadd741d4baed25ca16cd4119 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:49:27.494055 14083 raft_consensus.cc:504] T 00000000000000000000000000000000 P 46c4e03eadd741d4baed25ca16cd4119 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46c4e03eadd741d4baed25ca16cd4119" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36633 } }
I20260619 14:49:27.494470 14083 leader_election.cc:302] T 00000000000000000000000000000000 P 46c4e03eadd741d4baed25ca16cd4119 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 46c4e03eadd741d4baed25ca16cd4119; no voters:
I20260619 14:49:27.494858 14083 leader_election.cc:288] T 00000000000000000000000000000000 P 46c4e03eadd741d4baed25ca16cd4119 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:49:27.494935 14088 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 46c4e03eadd741d4baed25ca16cd4119 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:49:27.495179 14088 raft_consensus.cc:686] T 00000000000000000000000000000000 P 46c4e03eadd741d4baed25ca16cd4119 [term 1 LEADER]: Becoming Leader. State: Replica: 46c4e03eadd741d4baed25ca16cd4119, State: Running, Role: LEADER
I20260619 14:49:27.495579 14088 consensus_queue.cc:237] T 00000000000000000000000000000000 P 46c4e03eadd741d4baed25ca16cd4119 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46c4e03eadd741d4baed25ca16cd4119" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36633 } }
I20260619 14:49:27.495712 14086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:27.479281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57526 (local address 127.0.98.62:36633)
0619 14:49:27.479776 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:27.479797 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:27.479841 (+ 44us) server_negotiation.cc:407] Connection header received
0619 14:49:27.480766 (+ 925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:27.480792 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:27.481302 (+ 510us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:27.481698 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:49:27.482519 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:27.484015 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:27.485165 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:27.485831 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:27.486010 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:27.488925 (+ 2915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:49:27.488932 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:49:27.488953 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:49:27.488986 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:49:27.491817 (+ 2831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:27.492547 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:27.492553 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:27.492559 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:27.492734 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:27.493097 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:27.493101 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:27.493102 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:27.493622 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:49:27.493860 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:27.494172 (+ 312us) server_negotiation.cc:299] Negotiation successful
0619 14:49:27.494495 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":146,"threads_started":1}
I20260619 14:49:27.496397 14083 sys_catalog.cc:564] T 00000000000000000000000000000000 P 46c4e03eadd741d4baed25ca16cd4119 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:49:27.498170 14089 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46c4e03eadd741d4baed25ca16cd4119 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "46c4e03eadd741d4baed25ca16cd4119" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46c4e03eadd741d4baed25ca16cd4119" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36633 } } }
I20260619 14:49:27.498425 14089 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46c4e03eadd741d4baed25ca16cd4119 [sys.catalog]: This master's current role is: LEADER
I20260619 14:49:27.498395 14097 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46c4e03eadd741d4baed25ca16cd4119 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46c4e03eadd741d4baed25ca16cd4119. Latest consensus state: current_term: 1 leader_uuid: "46c4e03eadd741d4baed25ca16cd4119" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46c4e03eadd741d4baed25ca16cd4119" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36633 } } }
I20260619 14:49:27.498621 14097 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46c4e03eadd741d4baed25ca16cd4119 [sys.catalog]: This master's current role is: LEADER
I20260619 14:49:27.499152 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260619 14:49:27.502271 14103 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 46c4e03eadd741d4baed25ca16cd4119: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:49:27.502377 14103 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:49:27.502542 14104 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:49:27.503314 14104 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:49:27.509722 14104 catalog_manager.cc:1347] Generated new cluster ID: f90500ff8fa749c6adf6eaed62f25b60
I20260619 14:49:27.509816 14104 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:49:27.524036 14104 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:49:27.525432 14104 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:49:27.543535 14104 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 46c4e03eadd741d4baed25ca16cd4119: Generated new TSK 0
I20260619 14:49:27.544457 14104 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:49:27.555836 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 14015
2026-06-19T14:49:27Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1564 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:49:27 dist-test-slave-2sb1 krb5kdc[14111](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:49:27 dist-test-slave-2sb1 krb5kdc[14111](info): set up 2 sockets
Jun 19 14:49:27 dist-test-slave-2sb1 krb5kdc[14111](info): commencing operation
krb5kdc: starting...
W20260619 14:49:28.681102 392 mini_kdc.cc:121] Time spent starting KDC: real 1.104s user 0.006s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/krb5kdc/test-user.keytab.
Jun 19 14:49:28 dist-test-slave-2sb1 krb5kdc[14111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880568, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:49:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:49:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 14:49:28.832553 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38293
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38293
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:49:28.957852 14124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:49:28.958231 14124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:49:28.958319 14124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:49:28.963275 14124 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:49:28.963423 14124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:49:28.963462 14124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:49:28.963505 14124 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:49:28.963541 14124 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:49:28.969606 14124 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38293
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38293
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14124
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:49:28.970880 14124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:49:28.972057 14124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:49:28.981686 14130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:49:28.981686 14129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:49:28.981690 14132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:49:28.981962 14124 server_base.cc:1034] running on GCE node
I20260619 14:49:28.982653 14124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:49:28.983853 14124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:49:28.985066 14124 hybrid_clock.cc:648] HybridClock initialized: now 1781880568985018 us; error 65 us; skew 500 ppm
Jun 19 14:49:28 dist-test-slave-2sb1 krb5kdc[14111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880568, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:49:28.988519 14124 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 14:49:28.989863 14124 webserver.cc:460] Webserver started at http://127.0.98.62:33531/ using document root <none> and password file <none>
I20260619 14:49:28.990433 14124 fs_manager.cc:359] Metadata directory not provided
I20260619 14:49:28.990496 14124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:49:28.990667 14124 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:49:28.992849 14124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/master-0/data/instance:
uuid: "b6771a03d5c94a049845292e361e9034"
format_stamp: "Formatted at 2026-06-19 14:49:28 on dist-test-slave-2sb1"
server_key: "9a323fb970da1d2676ccf25ffee35f3d"
server_key_iv: "742b667a3ad03cebfdddd1ad034f72b9"
server_key_version: "encryptionkey@0"
I20260619 14:49:28.993456 14124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/master-0/wal/instance:
uuid: "b6771a03d5c94a049845292e361e9034"
format_stamp: "Formatted at 2026-06-19 14:49:28 on dist-test-slave-2sb1"
server_key: "9a323fb970da1d2676ccf25ffee35f3d"
server_key_iv: "742b667a3ad03cebfdddd1ad034f72b9"
server_key_version: "encryptionkey@0"
I20260619 14:49:28.997591 14124 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 14:49:29.000473 14139 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:29.002076 14124 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 14:49:29.002251 14124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/master-0/wal
uuid: "b6771a03d5c94a049845292e361e9034"
format_stamp: "Formatted at 2026-06-19 14:49:28 on dist-test-slave-2sb1"
server_key: "9a323fb970da1d2676ccf25ffee35f3d"
server_key_iv: "742b667a3ad03cebfdddd1ad034f72b9"
server_key_version: "encryptionkey@0"
I20260619 14:49:29.002411 14124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:49:29.018972 14124 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:49:29.023023 14124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:49:29.023283 14124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:49:29.031819 14124 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38293
I20260619 14:49:29.032269 14191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38293 every 8 connection(s)
I20260619 14:49:29.033047 14124 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:49:29.036695 14192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:49:29.041282 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 14124
I20260619 14:49:29.041464 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/master-0/wal/instance
I20260619 14:49:29.041785 392 external_mini_cluster.cc:1428] Setting key b01815935af0370c5ce6d875d4c97517
I20260619 14:49:29.043686 14192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6771a03d5c94a049845292e361e9034: Bootstrap starting.
I20260619 14:49:29.046337 14192 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b6771a03d5c94a049845292e361e9034: Neither blocks nor log segments found. Creating new log.
I20260619 14:49:29.047319 14192 log.cc:824] T 00000000000000000000000000000000 P b6771a03d5c94a049845292e361e9034: Log is configured to *not* fsync() on all Append() calls
I20260619 14:49:29.051877 14192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6771a03d5c94a049845292e361e9034: No bootstrap required, opened a new log
Jun 19 14:49:29 dist-test-slave-2sb1 krb5kdc[14111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880568, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:49:29.057663 14192 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6771a03d5c94a049845292e361e9034 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6771a03d5c94a049845292e361e9034" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38293 } }
I20260619 14:49:29.058027 14192 raft_consensus.cc:374] T 00000000000000000000000000000000 P b6771a03d5c94a049845292e361e9034 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:49:29.058120 14192 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6771a03d5c94a049845292e361e9034 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6771a03d5c94a049845292e361e9034, State: Initialized, Role: FOLLOWER
I20260619 14:49:29.058763 14192 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6771a03d5c94a049845292e361e9034 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6771a03d5c94a049845292e361e9034" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38293 } }
I20260619 14:49:29.058974 14192 raft_consensus.cc:388] T 00000000000000000000000000000000 P b6771a03d5c94a049845292e361e9034 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:49:29.059088 14192 raft_consensus.cc:482] T 00000000000000000000000000000000 P b6771a03d5c94a049845292e361e9034 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:49:29.059226 14192 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6771a03d5c94a049845292e361e9034 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:49:29.060777 14192 raft_consensus.cc:504] T 00000000000000000000000000000000 P b6771a03d5c94a049845292e361e9034 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6771a03d5c94a049845292e361e9034" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38293 } }
I20260619 14:49:29.061285 14192 leader_election.cc:302] T 00000000000000000000000000000000 P b6771a03d5c94a049845292e361e9034 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b6771a03d5c94a049845292e361e9034; no voters:
I20260619 14:49:29.061699 14192 leader_election.cc:288] T 00000000000000000000000000000000 P b6771a03d5c94a049845292e361e9034 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:49:29.061825 14197 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b6771a03d5c94a049845292e361e9034 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:49:29.062403 14197 raft_consensus.cc:686] T 00000000000000000000000000000000 P b6771a03d5c94a049845292e361e9034 [term 1 LEADER]: Becoming Leader. State: Replica: b6771a03d5c94a049845292e361e9034, State: Running, Role: LEADER
I20260619 14:49:29.062827 14197 consensus_queue.cc:237] T 00000000000000000000000000000000 P b6771a03d5c94a049845292e361e9034 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6771a03d5c94a049845292e361e9034" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38293 } }
I20260619 14:49:29.063021 14195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:29.043542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37072 (local address 127.0.98.62:38293)
0619 14:49:29.044302 (+ 760us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:29.044321 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:29.044362 (+ 41us) server_negotiation.cc:407] Connection header received
0619 14:49:29.045138 (+ 776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:29.045159 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:29.045552 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:29.045873 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:49:29.046953 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:29.048365 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:29.049420 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:29.050112 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:29.050433 (+ 321us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:29.054264 (+ 3831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:49:29.054286 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:49:29.054312 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:49:29.054366 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:49:29.058163 (+ 3797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:29.059021 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:29.059031 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:29.059041 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:29.059234 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:29.059837 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:29.059841 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:29.059843 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:29.060405 (+ 562us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:49:29.060695 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:29.061128 (+ 433us) server_negotiation.cc:299] Negotiation successful
0619 14:49:29.061439 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":344,"threads_started":1}
I20260619 14:49:29.063366 14192 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6771a03d5c94a049845292e361e9034 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:49:29.064631 14199 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6771a03d5c94a049845292e361e9034 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6771a03d5c94a049845292e361e9034. Latest consensus state: current_term: 1 leader_uuid: "b6771a03d5c94a049845292e361e9034" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6771a03d5c94a049845292e361e9034" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38293 } } }
I20260619 14:49:29.064812 14199 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6771a03d5c94a049845292e361e9034 [sys.catalog]: This master's current role is: LEADER
I20260619 14:49:29.065156 14198 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6771a03d5c94a049845292e361e9034 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b6771a03d5c94a049845292e361e9034" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6771a03d5c94a049845292e361e9034" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38293 } } }
I20260619 14:49:29.065256 14198 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6771a03d5c94a049845292e361e9034 [sys.catalog]: This master's current role is: LEADER
I20260619 14:49:29.065410 14206 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:49:29.068702 14206 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:49:29.075124 14206 catalog_manager.cc:1347] Generated new cluster ID: 3dd5a58639204044aeadf22270a16d68
I20260619 14:49:29.075255 14206 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:49:29.102569 14206 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:49:29.103634 14206 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:49:29.116365 14206 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b6771a03d5c94a049845292e361e9034: Generated new TSK 0
I20260619 14:49:29.117383 14206 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 14:49:29.157522 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38293
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:49:29.287317 14218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:49:29.287703 14218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:49:29.287761 14218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:49:29.292781 14218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:49:29.292874 14218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:49:29.292944 14218 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 14:49:29.299309 14218 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14218
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:49:29.300441 14218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:49:29.301613 14218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:49:29.314167 14226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:49:29.314219 14223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:49:29.314219 14224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:49:29.314850 14218 server_base.cc:1034] running on GCE node
I20260619 14:49:29.315464 14218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:49:29.316210 14218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:49:29.317598 14218 hybrid_clock.cc:648] HybridClock initialized: now 1781880569317560 us; error 36 us; skew 500 ppm
Jun 19 14:49:29 dist-test-slave-2sb1 krb5kdc[14111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880569, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:49:29.321605 14218 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 14:49:29.322687 14218 webserver.cc:460] Webserver started at http://127.0.98.1:42965/ using document root <none> and password file <none>
I20260619 14:49:29.323325 14218 fs_manager.cc:359] Metadata directory not provided
I20260619 14:49:29.323431 14218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:49:29.323678 14218 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:49:29.325987 14218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-0/data/instance:
uuid: "756f220b41d0464b947550c9f879f31b"
format_stamp: "Formatted at 2026-06-19 14:49:29 on dist-test-slave-2sb1"
server_key: "d7ec8fd1d6cde41b65d369fc1d9e6ebc"
server_key_iv: "530e145e2ffdec8a5d393ce35f172ff2"
server_key_version: "encryptionkey@0"
I20260619 14:49:29.326612 14218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-0/wal/instance:
uuid: "756f220b41d0464b947550c9f879f31b"
format_stamp: "Formatted at 2026-06-19 14:49:29 on dist-test-slave-2sb1"
server_key: "d7ec8fd1d6cde41b65d369fc1d9e6ebc"
server_key_iv: "530e145e2ffdec8a5d393ce35f172ff2"
server_key_version: "encryptionkey@0"
I20260619 14:49:29.331156 14218 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 14:49:29.334173 14233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:29.335827 14218 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 14:49:29.335975 14218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-0/wal
uuid: "756f220b41d0464b947550c9f879f31b"
format_stamp: "Formatted at 2026-06-19 14:49:29 on dist-test-slave-2sb1"
server_key: "d7ec8fd1d6cde41b65d369fc1d9e6ebc"
server_key_iv: "530e145e2ffdec8a5d393ce35f172ff2"
server_key_version: "encryptionkey@0"
I20260619 14:49:29.336088 14218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:49:29.353981 14218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:49:29.358407 14218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:49:29.358593 14218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:49:29.359241 14218 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:49:29.360098 14218 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:49:29.360147 14218 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:29.360190 14218 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:49:29.360205 14218 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:29.371767 14218 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44465
I20260619 14:49:29.371924 14346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44465 every 8 connection(s)
I20260619 14:49:29.373015 14218 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-0/data/info.pb
I20260619 14:49:29.375670 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 14218
I20260619 14:49:29.375911 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-0/wal/instance
I20260619 14:49:29.376240 392 external_mini_cluster.cc:1428] Setting key fdc6a5fbfce7ce314ff943d637b44496
Jun 19 14:49:29 dist-test-slave-2sb1 krb5kdc[14111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880569, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 14:49:29.393462 14195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:29.375873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46315 (local address 127.0.98.62:38293)
0619 14:49:29.376105 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:29.376111 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:29.377065 (+ 954us) server_negotiation.cc:407] Connection header received
0619 14:49:29.378739 (+ 1674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:29.378742 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:29.378793 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:29.378890 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:49:29.380983 (+ 2093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:29.381546 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:29.383704 (+ 2158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:29.384057 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:29.384162 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:29.388030 (+ 3868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:49:29.388034 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:49:29.388036 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:49:29.388065 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:49:29.389803 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:29.390646 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:29.390649 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:29.390651 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:29.390750 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:29.391536 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:29.391539 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:29.391540 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:29.391761 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:49:29.392039 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:29.392916 (+ 877us) server_negotiation.cc:299] Negotiation successful
0619 14:49:29.393095 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125}
I20260619 14:49:29.394179 14349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:29.376318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38293 (local address 127.0.98.1:46315)
0619 14:49:29.376897 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:29.376961 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0619 14:49:29.378444 (+ 1483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:49:29.379094 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:49:29.379107 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:49:29.379707 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:49:29.380791 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:29.380813 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:29.381697 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:29.381704 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:29.383392 (+ 1688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:29.383408 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:29.384297 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:29.384303 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:29.384790 (+ 487us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:29.385567 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:49:29.385598 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:49:29.387786 (+ 2188us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:49:29.389954 (+ 2168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:49:29.389960 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:49:29.389983 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:49:29.390323 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:49:29.391066 (+ 743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:49:29.391069 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:49:29.391071 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:49:29.391358 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:49:29.392053 (+ 695us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:49:29.392065 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:49:29.392480 (+ 415us) client_negotiation.cc:769] Sending connection context
0619 14:49:29.392761 (+ 281us) client_negotiation.cc:240] Negotiation successful
0619 14:49:29.393139 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":143,"threads_started":1}
I20260619 14:49:29.396238 14347 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38293
I20260619 14:49:29.396612 14347 heartbeater.cc:464] Registering TS with master...
I20260619 14:49:29.397298 14347 heartbeater.cc:511] Master 127.0.98.62:38293 requested a full tablet report, sending...
I20260619 14:49:29.399381 14156 ts_manager.cc:194] Registered new tserver with Master: 756f220b41d0464b947550c9f879f31b (127.0.98.1:44465)
I20260619 14:49:29.401031 14156 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:46315
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 14:49:29.413676 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38293
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:49:29.532513 14352 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:49:29.532907 14352 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:49:29.532986 14352 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:49:29.537406 14352 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:49:29.537509 14352 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:49:29.537606 14352 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 14:49:29.543366 14352 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14352
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:49:29.544548 14352 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:49:29.545670 14352 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:49:29.557053 14357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:49:29.557204 14360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:49:29.557092 14352 server_base.cc:1034] running on GCE node
W20260619 14:49:29.557092 14358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:49:29.557782 14352 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:49:29.558609 14352 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:49:29.559803 14352 hybrid_clock.cc:648] HybridClock initialized: now 1781880569559772 us; error 53 us; skew 500 ppm
Jun 19 14:49:29 dist-test-slave-2sb1 krb5kdc[14111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880569, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:49:29.563668 14352 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 14:49:29.565171 14352 webserver.cc:460] Webserver started at http://127.0.98.2:33907/ using document root <none> and password file <none>
I20260619 14:49:29.565901 14352 fs_manager.cc:359] Metadata directory not provided
I20260619 14:49:29.565985 14352 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:49:29.566228 14352 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:49:29.568604 14352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-1/data/instance:
uuid: "7f8375b6be764013974e4e579f7a6e80"
format_stamp: "Formatted at 2026-06-19 14:49:29 on dist-test-slave-2sb1"
server_key: "75e11f0fa2159637d278bbc0f234423a"
server_key_iv: "dc2c124dcddeaee51cdca4d7b066e3dd"
server_key_version: "encryptionkey@0"
I20260619 14:49:29.569501 14352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-1/wal/instance:
uuid: "7f8375b6be764013974e4e579f7a6e80"
format_stamp: "Formatted at 2026-06-19 14:49:29 on dist-test-slave-2sb1"
server_key: "75e11f0fa2159637d278bbc0f234423a"
server_key_iv: "dc2c124dcddeaee51cdca4d7b066e3dd"
server_key_version: "encryptionkey@0"
I20260619 14:49:29.574126 14352 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 14:49:29.577191 14367 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:29.579187 14352 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 14:49:29.579370 14352 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-1/wal
uuid: "7f8375b6be764013974e4e579f7a6e80"
format_stamp: "Formatted at 2026-06-19 14:49:29 on dist-test-slave-2sb1"
server_key: "75e11f0fa2159637d278bbc0f234423a"
server_key_iv: "dc2c124dcddeaee51cdca4d7b066e3dd"
server_key_version: "encryptionkey@0"
I20260619 14:49:29.579517 14352 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:49:29.602721 14352 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:49:29.607167 14352 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:49:29.607594 14352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:49:29.608494 14352 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:49:29.609494 14352 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:49:29.609577 14352 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:29.609654 14352 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:49:29.609722 14352 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:29.621261 14352 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34133
I20260619 14:49:29.621335 14480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34133 every 8 connection(s)
I20260619 14:49:29.622357 14352 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-1/data/info.pb
I20260619 14:49:29.631979 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 14352
I20260619 14:49:29.632179 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-1/wal/instance
I20260619 14:49:29.632550 392 external_mini_cluster.cc:1428] Setting key 5fcb3525883fbc1df85291ead81e6810
Jun 19 14:49:29 dist-test-slave-2sb1 krb5kdc[14111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880569, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:49:29.638924 14195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:29.624723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55529 (local address 127.0.98.62:38293)
0619 14:49:29.624920 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:29.624925 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:29.625924 (+ 999us) server_negotiation.cc:407] Connection header received
0619 14:49:29.627486 (+ 1562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:29.627489 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:29.627546 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:29.627630 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:49:29.629152 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:29.629763 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:29.631166 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:29.631419 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:29.631512 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:29.634655 (+ 3143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:49:29.634658 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:49:29.634660 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:49:29.634686 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:49:29.636355 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:29.637067 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:29.637077 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:29.637079 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:29.637162 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:29.637560 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:29.637564 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:29.637566 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:29.637778 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:49:29.637896 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:29.638551 (+ 655us) server_negotiation.cc:299] Negotiation successful
0619 14:49:29.638714 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260619 14:49:29.639947 14483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:29.625179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38293 (local address 127.0.98.2:55529)
0619 14:49:29.625751 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:29.625797 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 14:49:29.627147 (+ 1350us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:49:29.627845 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:49:29.627853 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:49:29.628321 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:49:29.628985 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:29.629000 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:29.629932 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:29.629935 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:29.631022 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:29.631033 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:29.631604 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:29.631607 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:29.631795 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:29.632517 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:49:29.632534 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:49:29.634436 (+ 1902us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:49:29.636502 (+ 2066us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:49:29.636510 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:49:29.636537 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:49:29.636934 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:49:29.637269 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:49:29.637273 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:49:29.637275 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:49:29.637440 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:49:29.637909 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:49:29.637918 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:49:29.638252 (+ 334us) client_negotiation.cc:769] Sending connection context
0619 14:49:29.638486 (+ 234us) client_negotiation.cc:240] Negotiation successful
0619 14:49:29.638896 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":154,"threads_started":1}
I20260619 14:49:29.641533 14481 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38293
I20260619 14:49:29.641873 14481 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 14:49:29.642486 14481 heartbeater.cc:511] Master 127.0.98.62:38293 requested a full tablet report, sending...
I20260619 14:49:29.643819 14156 ts_manager.cc:194] Registered new tserver with Master: 7f8375b6be764013974e4e579f7a6e80 (127.0.98.2:34133)
I20260619 14:49:29.644501 14156 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55529
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 14:49:29.668217 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38293
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:49:29.795436 14486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:49:29.795755 14486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:49:29.795804 14486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:49:29.800139 14486 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:49:29.800232 14486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:49:29.800294 14486 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 14:49:29.805989 14486 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14486
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:49:29.807123 14486 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:49:29.808216 14486 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:49:29.820032 14491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:49:29.820086 14492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:49:29.820093 14494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:49:29.820142 14486 server_base.cc:1034] running on GCE node
I20260619 14:49:29.820905 14486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:49:29.821715 14486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:49:29.822980 14486 hybrid_clock.cc:648] HybridClock initialized: now 1781880569822933 us; error 66 us; skew 500 ppm
Jun 19 14:49:29 dist-test-slave-2sb1 krb5kdc[14111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880569, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:49:29.826701 14486 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 14:49:29.827768 14486 webserver.cc:460] Webserver started at http://127.0.98.3:44395/ using document root <none> and password file <none>
I20260619 14:49:29.828403 14486 fs_manager.cc:359] Metadata directory not provided
I20260619 14:49:29.828502 14486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:49:29.828807 14486 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:49:29.831581 14486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-2/data/instance:
uuid: "0cf4adf2a6664d31af185aeb05846b9d"
format_stamp: "Formatted at 2026-06-19 14:49:29 on dist-test-slave-2sb1"
server_key: "67bcf70d36ed12b1ba5463a5836affec"
server_key_iv: "057d4659b4200470957a6190fdac9a5d"
server_key_version: "encryptionkey@0"
I20260619 14:49:29.832288 14486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-2/wal/instance:
uuid: "0cf4adf2a6664d31af185aeb05846b9d"
format_stamp: "Formatted at 2026-06-19 14:49:29 on dist-test-slave-2sb1"
server_key: "67bcf70d36ed12b1ba5463a5836affec"
server_key_iv: "057d4659b4200470957a6190fdac9a5d"
server_key_version: "encryptionkey@0"
I20260619 14:49:29.837064 14486 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.002s
I20260619 14:49:29.840138 14501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:29.841946 14486 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:49:29.842124 14486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-2/wal
uuid: "0cf4adf2a6664d31af185aeb05846b9d"
format_stamp: "Formatted at 2026-06-19 14:49:29 on dist-test-slave-2sb1"
server_key: "67bcf70d36ed12b1ba5463a5836affec"
server_key_iv: "057d4659b4200470957a6190fdac9a5d"
server_key_version: "encryptionkey@0"
I20260619 14:49:29.842262 14486 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:49:29.853770 14486 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:49:29.857553 14486 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:49:29.857792 14486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:49:29.858426 14486 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:49:29.859321 14486 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:49:29.859395 14486 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:29.859463 14486 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:49:29.859506 14486 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:29.870493 14486 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33209
I20260619 14:49:29.870559 14614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33209 every 8 connection(s)
I20260619 14:49:29.871579 14486 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-2/data/info.pb
I20260619 14:49:29.876606 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 14486
I20260619 14:49:29.876784 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-2/wal/instance
I20260619 14:49:29.877086 392 external_mini_cluster.cc:1428] Setting key 4d96dd271cc7389b907e498fa940d5c6
Jun 19 14:49:29 dist-test-slave-2sb1 krb5kdc[14111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880569, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:49:29.889202 14195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:29.873895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41345 (local address 127.0.98.62:38293)
0619 14:49:29.874088 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:29.874092 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:29.875181 (+ 1089us) server_negotiation.cc:407] Connection header received
0619 14:49:29.876780 (+ 1599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:29.876783 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:29.876837 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:29.876953 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:49:29.878609 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:29.879486 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:29.881038 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:29.881301 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:29.881416 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:29.884602 (+ 3186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:49:29.884606 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:49:29.884608 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:49:29.884683 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:49:29.886468 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:29.887190 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:29.887193 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:29.887194 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:29.887289 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:29.887762 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:29.887765 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:29.887766 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:29.887994 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:49:29.888091 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:29.888941 (+ 850us) server_negotiation.cc:299] Negotiation successful
0619 14:49:29.889067 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260619 14:49:29.890213 14617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:29.874300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38293 (local address 127.0.98.3:41345)
0619 14:49:29.874999 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:29.875046 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 14:49:29.876454 (+ 1408us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:49:29.877200 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:49:29.877212 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:49:29.877677 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:49:29.878446 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:29.878463 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:29.879746 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:29.879750 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:29.880847 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:29.880859 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:29.881546 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:29.881549 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:29.881717 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:29.882522 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:49:29.882541 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:49:29.884367 (+ 1826us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:49:29.886682 (+ 2315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:49:29.886688 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:49:29.886703 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:49:29.887060 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:49:29.887480 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:49:29.887483 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:49:29.887484 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:49:29.887648 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:49:29.888108 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:49:29.888114 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:49:29.888406 (+ 292us) client_negotiation.cc:769] Sending connection context
0619 14:49:29.888598 (+ 192us) client_negotiation.cc:240] Negotiation successful
0619 14:49:29.889136 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":190,"threads_started":1}
I20260619 14:49:29.891961 14615 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38293
I20260619 14:49:29.892272 14615 heartbeater.cc:464] Registering TS with master...
I20260619 14:49:29.893006 14615 heartbeater.cc:511] Master 127.0.98.62:38293 requested a full tablet report, sending...
I20260619 14:49:29.894256 14154 ts_manager.cc:194] Registered new tserver with Master: 0cf4adf2a6664d31af185aeb05846b9d (127.0.98.3:33209)
I20260619 14:49:29.894995 14154 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41345
I20260619 14:49:29.905222 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:49:29.918083 14195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:29.907773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37074 (local address 127.0.98.62:38293)
0619 14:49:29.908007 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:29.908011 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:29.908308 (+ 297us) server_negotiation.cc:407] Connection header received
0619 14:49:29.908552 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:29.908556 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:29.908609 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:29.908734 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:49:29.909751 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:29.910477 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:29.911511 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:29.911758 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:29.911999 (+ 241us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:29.913879 (+ 1880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:49:29.913883 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:49:29.913886 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:49:29.913918 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:49:29.915720 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:29.916365 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:29.916369 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:29.916370 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:29.916460 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:29.916868 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:29.916871 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:29.916872 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:29.917161 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:49:29.917301 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:29.917646 (+ 345us) server_negotiation.cc:299] Negotiation successful
0619 14:49:29.917809 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156}
I20260619 14:49:29.922585 14154 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37074:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:49:29.926162 14154 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:49:29.943851 14627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:29.937035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38136 (local address 127.0.98.1:44465)
0619 14:49:29.937464 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:29.937471 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:29.937873 (+ 402us) server_negotiation.cc:407] Connection header received
0619 14:49:29.938119 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:29.938124 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:29.938348 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:29.938506 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:49:29.939093 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:29.940367 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:29.942091 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:29.942792 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:29.942900 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:29.943054 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:29.943532 (+ 478us) server_negotiation.cc:299] Negotiation successful
0619 14:49:29.943643 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":85,"threads_started":1}
I20260619 14:49:29.944118 14630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:29.937808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44580 (local address 127.0.98.3:33209)
0619 14:49:29.938854 (+ 1046us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:29.938865 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:29.938887 (+ 22us) server_negotiation.cc:407] Connection header received
0619 14:49:29.938959 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:29.938964 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:29.939158 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:29.939352 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:49:29.939593 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:29.940693 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:29.942548 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:29.943378 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:29.943498 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:29.943629 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:29.943788 (+ 159us) server_negotiation.cc:299] Negotiation successful
0619 14:49:29.943901 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":942,"thread_start_us":143,"threads_started":1}
I20260619 14:49:29.944350 14626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:29.937082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33209 (local address 127.0.0.1:44580)
0619 14:49:29.938033 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:29.938049 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:49:29.938132 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:49:29.939313 (+ 1181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:49:29.939316 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:49:29.939331 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:49:29.939477 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:29.939484 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:29.940988 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:29.940991 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:29.942396 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:29.942407 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:29.943524 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:29.943528 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:29.943640 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:29.943666 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 14:49:29.943758 (+ 92us) client_negotiation.cc:240] Negotiation successful
0619 14:49:29.943901 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":878,"thread_start_us":95,"threads_started":1}
I20260619 14:49:29.944571 14631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:29.938221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58538 (local address 127.0.98.2:34133)
0619 14:49:29.938892 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:29.938907 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:29.939131 (+ 224us) server_negotiation.cc:407] Connection header received
0619 14:49:29.939284 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:29.939291 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:29.939476 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:29.939624 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:49:29.940056 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:29.941358 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:29.942861 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:29.943529 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:29.943640 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:29.943794 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:29.944276 (+ 482us) server_negotiation.cc:299] Negotiation successful
0619 14:49:29.944390 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":265,"threads_started":1}
I20260619 14:49:29.944866 14628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:29.937266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34133 (local address 127.0.0.1:58538)
0619 14:49:29.939021 (+ 1755us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:29.939033 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:49:29.939163 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:49:29.939820 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:49:29.939823 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:49:29.939840 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:49:29.939937 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:29.939941 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:29.941524 (+ 1583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:29.941527 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:29.942735 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:29.942742 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:29.943859 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:29.943862 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:29.943968 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:29.944000 (+ 32us) client_negotiation.cc:769] Sending connection context
0619 14:49:29.944075 (+ 75us) client_negotiation.cc:240] Negotiation successful
0619 14:49:29.944141 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1686,"thread_start_us":101,"threads_started":1}
I20260619 14:49:29.944918 14629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:29.936921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44465 (local address 127.0.0.1:38136)
0619 14:49:29.937699 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:29.937755 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 14:49:29.937917 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:49:29.938759 (+ 842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:49:29.938765 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:49:29.938790 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:49:29.938962 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:29.938969 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:29.940584 (+ 1615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:29.940587 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:29.941958 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:29.941968 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:29.943159 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:29.943163 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:29.943294 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:29.943347 (+ 53us) client_negotiation.cc:769] Sending connection context
0619 14:49:29.944035 (+ 688us) client_negotiation.cc:240] Negotiation successful
0619 14:49:29.944116 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"mutex_wait_us":1,"thread_start_us":121,"threads_started":1}
I20260619 14:49:29.948050 14281 tablet_service.cc:1467] Processing CreateTablet for tablet fd0255808a824cafb15d5e1a2b922acf (DEFAULT_TABLE table=test-table [id=13d295a8621d48e59bc208f1dc079096]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:49:29.948050 14549 tablet_service.cc:1467] Processing CreateTablet for tablet fd0255808a824cafb15d5e1a2b922acf (DEFAULT_TABLE table=test-table [id=13d295a8621d48e59bc208f1dc079096]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:49:29.948053 14415 tablet_service.cc:1467] Processing CreateTablet for tablet fd0255808a824cafb15d5e1a2b922acf (DEFAULT_TABLE table=test-table [id=13d295a8621d48e59bc208f1dc079096]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:49:29.949247 14549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd0255808a824cafb15d5e1a2b922acf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:49:29.949244 14281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd0255808a824cafb15d5e1a2b922acf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:49:29.949245 14415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd0255808a824cafb15d5e1a2b922acf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:49:29.956176 14632 tablet_bootstrap.cc:492] T fd0255808a824cafb15d5e1a2b922acf P 7f8375b6be764013974e4e579f7a6e80: Bootstrap starting.
I20260619 14:49:29.957419 14634 tablet_bootstrap.cc:492] T fd0255808a824cafb15d5e1a2b922acf P 756f220b41d0464b947550c9f879f31b: Bootstrap starting.
I20260619 14:49:29.959465 14634 tablet_bootstrap.cc:654] T fd0255808a824cafb15d5e1a2b922acf P 756f220b41d0464b947550c9f879f31b: Neither blocks nor log segments found. Creating new log.
I20260619 14:49:29.959841 14632 tablet_bootstrap.cc:654] T fd0255808a824cafb15d5e1a2b922acf P 7f8375b6be764013974e4e579f7a6e80: Neither blocks nor log segments found. Creating new log.
I20260619 14:49:29.960342 14634 log.cc:824] T fd0255808a824cafb15d5e1a2b922acf P 756f220b41d0464b947550c9f879f31b: Log is configured to *not* fsync() on all Append() calls
I20260619 14:49:29.960738 14632 log.cc:824] T fd0255808a824cafb15d5e1a2b922acf P 7f8375b6be764013974e4e579f7a6e80: Log is configured to *not* fsync() on all Append() calls
I20260619 14:49:29.961058 14633 tablet_bootstrap.cc:492] T fd0255808a824cafb15d5e1a2b922acf P 0cf4adf2a6664d31af185aeb05846b9d: Bootstrap starting.
I20260619 14:49:29.962792 14634 tablet_bootstrap.cc:492] T fd0255808a824cafb15d5e1a2b922acf P 756f220b41d0464b947550c9f879f31b: No bootstrap required, opened a new log
I20260619 14:49:29.963135 14634 ts_tablet_manager.cc:1397] T fd0255808a824cafb15d5e1a2b922acf P 756f220b41d0464b947550c9f879f31b: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 14:49:29.963212 14632 tablet_bootstrap.cc:492] T fd0255808a824cafb15d5e1a2b922acf P 7f8375b6be764013974e4e579f7a6e80: No bootstrap required, opened a new log
I20260619 14:49:29.963454 14632 ts_tablet_manager.cc:1397] T fd0255808a824cafb15d5e1a2b922acf P 7f8375b6be764013974e4e579f7a6e80: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260619 14:49:29.963765 14633 tablet_bootstrap.cc:654] T fd0255808a824cafb15d5e1a2b922acf P 0cf4adf2a6664d31af185aeb05846b9d: Neither blocks nor log segments found. Creating new log.
I20260619 14:49:29.965113 14633 log.cc:824] T fd0255808a824cafb15d5e1a2b922acf P 0cf4adf2a6664d31af185aeb05846b9d: Log is configured to *not* fsync() on all Append() calls
I20260619 14:49:29.967600 14632 raft_consensus.cc:348] T fd0255808a824cafb15d5e1a2b922acf P 7f8375b6be764013974e4e579f7a6e80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "756f220b41d0464b947550c9f879f31b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44465 } } peers { permanent_uuid: "0cf4adf2a6664d31af185aeb05846b9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33209 } } peers { permanent_uuid: "7f8375b6be764013974e4e579f7a6e80" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34133 } }
I20260619 14:49:29.967913 14632 raft_consensus.cc:374] T fd0255808a824cafb15d5e1a2b922acf P 7f8375b6be764013974e4e579f7a6e80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:49:29.967998 14632 raft_consensus.cc:729] T fd0255808a824cafb15d5e1a2b922acf P 7f8375b6be764013974e4e579f7a6e80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f8375b6be764013974e4e579f7a6e80, State: Initialized, Role: FOLLOWER
I20260619 14:49:29.968179 14633 tablet_bootstrap.cc:492] T fd0255808a824cafb15d5e1a2b922acf P 0cf4adf2a6664d31af185aeb05846b9d: No bootstrap required, opened a new log
I20260619 14:49:29.968652 14633 ts_tablet_manager.cc:1397] T fd0255808a824cafb15d5e1a2b922acf P 0cf4adf2a6664d31af185aeb05846b9d: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260619 14:49:29.968710 14632 consensus_queue.cc:260] T fd0255808a824cafb15d5e1a2b922acf P 7f8375b6be764013974e4e579f7a6e80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "756f220b41d0464b947550c9f879f31b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44465 } } peers { permanent_uuid: "0cf4adf2a6664d31af185aeb05846b9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33209 } } peers { permanent_uuid: "7f8375b6be764013974e4e579f7a6e80" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34133 } }
I20260619 14:49:29.968889 14634 raft_consensus.cc:348] T fd0255808a824cafb15d5e1a2b922acf P 756f220b41d0464b947550c9f879f31b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "756f220b41d0464b947550c9f879f31b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44465 } } peers { permanent_uuid: "0cf4adf2a6664d31af185aeb05846b9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33209 } } peers { permanent_uuid: "7f8375b6be764013974e4e579f7a6e80" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34133 } }
I20260619 14:49:29.969178 14634 raft_consensus.cc:374] T fd0255808a824cafb15d5e1a2b922acf P 756f220b41d0464b947550c9f879f31b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:49:29.969293 14634 raft_consensus.cc:729] T fd0255808a824cafb15d5e1a2b922acf P 756f220b41d0464b947550c9f879f31b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 756f220b41d0464b947550c9f879f31b, State: Initialized, Role: FOLLOWER
I20260619 14:49:29.969911 14632 ts_tablet_manager.cc:1428] T fd0255808a824cafb15d5e1a2b922acf P 7f8375b6be764013974e4e579f7a6e80: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 14:49:29.969887 14634 consensus_queue.cc:260] T fd0255808a824cafb15d5e1a2b922acf P 756f220b41d0464b947550c9f879f31b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "756f220b41d0464b947550c9f879f31b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44465 } } peers { permanent_uuid: "0cf4adf2a6664d31af185aeb05846b9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33209 } } peers { permanent_uuid: "7f8375b6be764013974e4e579f7a6e80" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34133 } }
I20260619 14:49:29.970352 14481 heartbeater.cc:503] Master 127.0.98.62:38293 was elected leader, sending a full tablet report...
I20260619 14:49:29.970888 14634 ts_tablet_manager.cc:1428] T fd0255808a824cafb15d5e1a2b922acf P 756f220b41d0464b947550c9f879f31b: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260619 14:49:29.971212 14347 heartbeater.cc:503] Master 127.0.98.62:38293 was elected leader, sending a full tablet report...
I20260619 14:49:29.973145 14633 raft_consensus.cc:348] T fd0255808a824cafb15d5e1a2b922acf P 0cf4adf2a6664d31af185aeb05846b9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "756f220b41d0464b947550c9f879f31b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44465 } } peers { permanent_uuid: "0cf4adf2a6664d31af185aeb05846b9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33209 } } peers { permanent_uuid: "7f8375b6be764013974e4e579f7a6e80" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34133 } }
I20260619 14:49:29.973454 14633 raft_consensus.cc:374] T fd0255808a824cafb15d5e1a2b922acf P 0cf4adf2a6664d31af185aeb05846b9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:49:29.973523 14633 raft_consensus.cc:729] T fd0255808a824cafb15d5e1a2b922acf P 0cf4adf2a6664d31af185aeb05846b9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cf4adf2a6664d31af185aeb05846b9d, State: Initialized, Role: FOLLOWER
I20260619 14:49:29.974128 14633 consensus_queue.cc:260] T fd0255808a824cafb15d5e1a2b922acf P 0cf4adf2a6664d31af185aeb05846b9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "756f220b41d0464b947550c9f879f31b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44465 } } peers { permanent_uuid: "0cf4adf2a6664d31af185aeb05846b9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33209 } } peers { permanent_uuid: "7f8375b6be764013974e4e579f7a6e80" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34133 } }
I20260619 14:49:29.975227 14615 heartbeater.cc:503] Master 127.0.98.62:38293 was elected leader, sending a full tablet report...
I20260619 14:49:29.975575 14633 ts_tablet_manager.cc:1428] T fd0255808a824cafb15d5e1a2b922acf P 0cf4adf2a6664d31af185aeb05846b9d: Time spent starting tablet: real 0.007s user 0.004s sys 0.002s
I20260619 14:49:30.012467 14638 raft_consensus.cc:482] T fd0255808a824cafb15d5e1a2b922acf P 7f8375b6be764013974e4e579f7a6e80 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:49:30.012737 14638 raft_consensus.cc:504] T fd0255808a824cafb15d5e1a2b922acf P 7f8375b6be764013974e4e579f7a6e80 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "756f220b41d0464b947550c9f879f31b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44465 } } peers { permanent_uuid: "0cf4adf2a6664d31af185aeb05846b9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33209 } } peers { permanent_uuid: "7f8375b6be764013974e4e579f7a6e80" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34133 } }
I20260619 14:49:30.014286 14638 leader_election.cc:288] T fd0255808a824cafb15d5e1a2b922acf P 7f8375b6be764013974e4e579f7a6e80 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 756f220b41d0464b947550c9f879f31b (127.0.98.1:44465), 0cf4adf2a6664d31af185aeb05846b9d (127.0.98.3:33209)
I20260619 14:49:30.020601 14630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:30.015132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57383 (local address 127.0.98.3:33209)
0619 14:49:30.015328 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:30.015333 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:30.015654 (+ 321us) server_negotiation.cc:407] Connection header received
0619 14:49:30.015758 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:30.015761 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:30.015819 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:30.016089 (+ 270us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:49:30.016470 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:30.017170 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:30.019236 (+ 2066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:30.019901 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:30.020000 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:30.020069 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:30.020351 (+ 282us) server_negotiation.cc:299] Negotiation successful
0619 14:49:30.020449 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 14:49:30.020601 14483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:30.014479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44465 (local address 127.0.98.2:42393)
0619 14:49:30.014633 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:30.014661 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0619 14:49:30.014796 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:49:30.015442 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:49:30.015446 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:49:30.015471 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:49:30.015613 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:30.015624 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:30.016968 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:30.016973 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:30.018842 (+ 1869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:30.018858 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:30.020008 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:30.020013 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:30.020270 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:30.020301 (+ 31us) client_negotiation.cc:769] Sending connection context
0619 14:49:30.020368 (+ 67us) client_negotiation.cc:240] Negotiation successful
0619 14:49:30.020428 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":179}
I20260619 14:49:30.020609 14641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:30.015090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33209 (local address 127.0.98.2:57383)
0619 14:49:30.015531 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:30.015545 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:49:30.015663 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:49:30.016148 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:49:30.016151 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:49:30.016175 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:49:30.016328 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:30.016335 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:30.017294 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:30.017297 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:30.019087 (+ 1790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:30.019095 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:30.020028 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:30.020031 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:30.020187 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:30.020217 (+ 30us) client_negotiation.cc:769] Sending connection context
0619 14:49:30.020318 (+ 101us) client_negotiation.cc:240] Negotiation successful
0619 14:49:30.020417 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378,"mutex_wait_us":143,"thread_start_us":128,"threads_started":1}
I20260619 14:49:30.021245 14627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:30.014945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42393 (local address 127.0.98.1:44465)
0619 14:49:30.015109 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:30.015116 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:30.015134 (+ 18us) server_negotiation.cc:407] Connection header received
0619 14:49:30.015191 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:30.015194 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:30.015252 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:30.015339 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:49:30.015752 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:30.016769 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:30.019036 (+ 2267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:30.019786 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:30.019919 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:30.020012 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:30.021023 (+ 1011us) server_negotiation.cc:299] Negotiation successful
0619 14:49:30.021128 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 14:49:30.021879 14301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd0255808a824cafb15d5e1a2b922acf" candidate_uuid: "7f8375b6be764013974e4e579f7a6e80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "756f220b41d0464b947550c9f879f31b" is_pre_election: true
I20260619 14:49:30.021891 14569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd0255808a824cafb15d5e1a2b922acf" candidate_uuid: "7f8375b6be764013974e4e579f7a6e80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cf4adf2a6664d31af185aeb05846b9d" is_pre_election: true
I20260619 14:49:30.022231 14301 raft_consensus.cc:2454] T fd0255808a824cafb15d5e1a2b922acf P 756f220b41d0464b947550c9f879f31b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f8375b6be764013974e4e579f7a6e80 in term 0.
I20260619 14:49:30.022231 14569 raft_consensus.cc:2454] T fd0255808a824cafb15d5e1a2b922acf P 0cf4adf2a6664d31af185aeb05846b9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f8375b6be764013974e4e579f7a6e80 in term 0.
I20260619 14:49:30.022797 14368 leader_election.cc:302] T fd0255808a824cafb15d5e1a2b922acf P 7f8375b6be764013974e4e579f7a6e80 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0cf4adf2a6664d31af185aeb05846b9d, 756f220b41d0464b947550c9f879f31b, 7f8375b6be764013974e4e579f7a6e80; no voters:
I20260619 14:49:30.023089 14638 raft_consensus.cc:2781] T fd0255808a824cafb15d5e1a2b922acf P 7f8375b6be764013974e4e579f7a6e80 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:49:30.023175 14638 raft_consensus.cc:482] T fd0255808a824cafb15d5e1a2b922acf P 7f8375b6be764013974e4e579f7a6e80 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:49:30.023218 14638 raft_consensus.cc:3037] T fd0255808a824cafb15d5e1a2b922acf P 7f8375b6be764013974e4e579f7a6e80 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:49:30.024333 14638 raft_consensus.cc:504] T fd0255808a824cafb15d5e1a2b922acf P 7f8375b6be764013974e4e579f7a6e80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "756f220b41d0464b947550c9f879f31b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44465 } } peers { permanent_uuid: "0cf4adf2a6664d31af185aeb05846b9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33209 } } peers { permanent_uuid: "7f8375b6be764013974e4e579f7a6e80" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34133 } }
I20260619 14:49:30.024791 14638 leader_election.cc:288] T fd0255808a824cafb15d5e1a2b922acf P 7f8375b6be764013974e4e579f7a6e80 [CANDIDATE]: Term 1 election: Requested vote from peers 756f220b41d0464b947550c9f879f31b (127.0.98.1:44465), 0cf4adf2a6664d31af185aeb05846b9d (127.0.98.3:33209)
I20260619 14:49:30.025244 14301 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd0255808a824cafb15d5e1a2b922acf" candidate_uuid: "7f8375b6be764013974e4e579f7a6e80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "756f220b41d0464b947550c9f879f31b"
I20260619 14:49:30.025233 14569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd0255808a824cafb15d5e1a2b922acf" candidate_uuid: "7f8375b6be764013974e4e579f7a6e80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cf4adf2a6664d31af185aeb05846b9d"
I20260619 14:49:30.025417 14301 raft_consensus.cc:3037] T fd0255808a824cafb15d5e1a2b922acf P 756f220b41d0464b947550c9f879f31b [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:49:30.025456 14569 raft_consensus.cc:3037] T fd0255808a824cafb15d5e1a2b922acf P 0cf4adf2a6664d31af185aeb05846b9d [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:49:30.026954 14569 raft_consensus.cc:2454] T fd0255808a824cafb15d5e1a2b922acf P 0cf4adf2a6664d31af185aeb05846b9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f8375b6be764013974e4e579f7a6e80 in term 1.
I20260619 14:49:30.027045 14301 raft_consensus.cc:2454] T fd0255808a824cafb15d5e1a2b922acf P 756f220b41d0464b947550c9f879f31b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f8375b6be764013974e4e579f7a6e80 in term 1.
I20260619 14:49:30.027442 14371 leader_election.cc:302] T fd0255808a824cafb15d5e1a2b922acf P 7f8375b6be764013974e4e579f7a6e80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 756f220b41d0464b947550c9f879f31b, 7f8375b6be764013974e4e579f7a6e80; no voters:
I20260619 14:49:30.027755 14638 raft_consensus.cc:2781] T fd0255808a824cafb15d5e1a2b922acf P 7f8375b6be764013974e4e579f7a6e80 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:49:30.027978 14638 raft_consensus.cc:686] T fd0255808a824cafb15d5e1a2b922acf P 7f8375b6be764013974e4e579f7a6e80 [term 1 LEADER]: Becoming Leader. State: Replica: 7f8375b6be764013974e4e579f7a6e80, State: Running, Role: LEADER
I20260619 14:49:30.028347 14638 consensus_queue.cc:237] T fd0255808a824cafb15d5e1a2b922acf P 7f8375b6be764013974e4e579f7a6e80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "756f220b41d0464b947550c9f879f31b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44465 } } peers { permanent_uuid: "0cf4adf2a6664d31af185aeb05846b9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33209 } } peers { permanent_uuid: "7f8375b6be764013974e4e579f7a6e80" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34133 } }
I20260619 14:49:30.032394 14156 catalog_manager.cc:5495] T fd0255808a824cafb15d5e1a2b922acf P 7f8375b6be764013974e4e579f7a6e80 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7f8375b6be764013974e4e579f7a6e80 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "7f8375b6be764013974e4e579f7a6e80" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "756f220b41d0464b947550c9f879f31b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cf4adf2a6664d31af185aeb05846b9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f8375b6be764013974e4e579f7a6e80" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34133 } health_report { overall_health: HEALTHY } } }
I20260619 14:49:30.051406 14631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:30.047062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58544 (local address 127.0.98.2:34133)
0619 14:49:30.047217 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:30.047221 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:30.047240 (+ 19us) server_negotiation.cc:407] Connection header received
0619 14:49:30.047354 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:30.047357 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:30.047415 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:30.047541 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:49:30.047835 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:30.048395 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:30.049481 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:30.049729 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:30.049856 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:30.050205 (+ 349us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:49:30.050918 (+ 713us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:49:30.051068 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:30.051135 (+ 67us) server_negotiation.cc:299] Negotiation successful
0619 14:49:30.051204 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260619 14:49:30.059006 14301 raft_consensus.cc:1261] T fd0255808a824cafb15d5e1a2b922acf P 756f220b41d0464b947550c9f879f31b [term 1 FOLLOWER]: Refusing update from remote peer 7f8375b6be764013974e4e579f7a6e80: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:49:30.059037 14569 raft_consensus.cc:1261] T fd0255808a824cafb15d5e1a2b922acf P 0cf4adf2a6664d31af185aeb05846b9d [term 1 FOLLOWER]: Refusing update from remote peer 7f8375b6be764013974e4e579f7a6e80: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:49:30.059957 14642 consensus_queue.cc:1035] T fd0255808a824cafb15d5e1a2b922acf P 7f8375b6be764013974e4e579f7a6e80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cf4adf2a6664d31af185aeb05846b9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:49:30.060267 14638 consensus_queue.cc:1035] T fd0255808a824cafb15d5e1a2b922acf P 7f8375b6be764013974e4e579f7a6e80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "756f220b41d0464b947550c9f879f31b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:49:30.070363 14646 mvcc.cc:205] Tried to move back new op lower bound from 7298582814953476096 to 7298582814839435264. Current Snapshot: MvccSnapshot[applied={T|T < 7298582814953476096}]
I20260619 14:49:30.071144 14647 mvcc.cc:205] Tried to move back new op lower bound from 7298582814953476096 to 7298582814839435264. Current Snapshot: MvccSnapshot[applied={T|T < 7298582814953476096}]
I20260619 14:49:30.078992 14156 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37074:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:49:30.079162 14156 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37074:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:49:30.082141 14156 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b6771a03d5c94a049845292e361e9034: Sending DeleteTablet for 3 replicas of tablet fd0255808a824cafb15d5e1a2b922acf
I20260619 14:49:30.083180 14415 tablet_service.cc:1514] Processing DeleteTablet for tablet fd0255808a824cafb15d5e1a2b922acf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:49:30 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58538
I20260619 14:49:30.083178 14549 tablet_service.cc:1514] Processing DeleteTablet for tablet fd0255808a824cafb15d5e1a2b922acf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:49:30 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44580
I20260619 14:49:30.083227 14281 tablet_service.cc:1514] Processing DeleteTablet for tablet fd0255808a824cafb15d5e1a2b922acf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:49:30 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38136
I20260619 14:49:30.083812 14656 tablet_replica.cc:330] T fd0255808a824cafb15d5e1a2b922acf P 0cf4adf2a6664d31af185aeb05846b9d: stopping tablet replica
I20260619 14:49:30.084102 14657 tablet_replica.cc:330] T fd0255808a824cafb15d5e1a2b922acf P 756f220b41d0464b947550c9f879f31b: stopping tablet replica
I20260619 14:49:30.084357 14655 tablet_replica.cc:330] T fd0255808a824cafb15d5e1a2b922acf P 7f8375b6be764013974e4e579f7a6e80: stopping tablet replica
I20260619 14:49:30.084726 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 14218
I20260619 14:49:30.084996 14657 raft_consensus.cc:2229] T fd0255808a824cafb15d5e1a2b922acf P 756f220b41d0464b947550c9f879f31b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:49:30.084772 14655 raft_consensus.cc:2229] T fd0255808a824cafb15d5e1a2b922acf P 7f8375b6be764013974e4e579f7a6e80 [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:49:30.086333 14655 raft_consensus.cc:2258] T fd0255808a824cafb15d5e1a2b922acf P 7f8375b6be764013974e4e579f7a6e80 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:49:30.088332 14655 ts_tablet_manager.cc:1905] T fd0255808a824cafb15d5e1a2b922acf P 7f8375b6be764013974e4e579f7a6e80: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:49:30.088743 14656 raft_consensus.cc:2229] T fd0255808a824cafb15d5e1a2b922acf P 0cf4adf2a6664d31af185aeb05846b9d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:49:30.089221 14656 raft_consensus.cc:2258] T fd0255808a824cafb15d5e1a2b922acf P 0cf4adf2a6664d31af185aeb05846b9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:49:30.090962 14656 ts_tablet_manager.cc:1905] T fd0255808a824cafb15d5e1a2b922acf P 0cf4adf2a6664d31af185aeb05846b9d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:49:30.092744 14655 ts_tablet_manager.cc:1918] T fd0255808a824cafb15d5e1a2b922acf P 7f8375b6be764013974e4e579f7a6e80: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:49:30.092967 14655 log.cc:1196] T fd0255808a824cafb15d5e1a2b922acf P 7f8375b6be764013974e4e579f7a6e80: Deleting WAL directory at /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-1/wal/wals/fd0255808a824cafb15d5e1a2b922acf
I20260619 14:49:30.093428 14655 ts_tablet_manager.cc:1939] T fd0255808a824cafb15d5e1a2b922acf P 7f8375b6be764013974e4e579f7a6e80: Deleting consensus metadata
I20260619 14:49:30.094121 14656 ts_tablet_manager.cc:1918] T fd0255808a824cafb15d5e1a2b922acf P 0cf4adf2a6664d31af185aeb05846b9d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:49:30.094342 14656 log.cc:1196] T fd0255808a824cafb15d5e1a2b922acf P 0cf4adf2a6664d31af185aeb05846b9d: Deleting WAL directory at /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-2/wal/wals/fd0255808a824cafb15d5e1a2b922acf
I20260619 14:49:30.094816 14656 ts_tablet_manager.cc:1939] T fd0255808a824cafb15d5e1a2b922acf P 0cf4adf2a6664d31af185aeb05846b9d: Deleting consensus metadata
W20260619 14:49:30.094821 14143 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44465 (error 108)
I20260619 14:49:30.095028 14141 catalog_manager.cc:4841] TS 7f8375b6be764013974e4e579f7a6e80 (127.0.98.2:34133): tablet fd0255808a824cafb15d5e1a2b922acf (table test-table [id=13d295a8621d48e59bc208f1dc079096]) successfully deleted
I20260619 14:49:30.095902 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 14352
I20260619 14:49:30.096820 14626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:30.095789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44465 (local address 127.0.0.1:38138)
0619 14:49:30.096558 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:30.096694 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":659}
W20260619 14:49:30.098523 14143 catalog_manager.cc:4584] TS 756f220b41d0464b947550c9f879f31b (127.0.98.1:44465): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fd0255808a824cafb15d5e1a2b922acf: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44465: connect: Connection refused (error 111)
I20260619 14:49:30.099025 14140 catalog_manager.cc:4841] TS 0cf4adf2a6664d31af185aeb05846b9d (127.0.98.3:33209): tablet fd0255808a824cafb15d5e1a2b922acf (table test-table [id=13d295a8621d48e59bc208f1dc079096]) successfully deleted
I20260619 14:49:30.104395 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 14486
I20260619 14:49:30.110656 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 14124
I20260619 14:49:30.118196 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38293
--webserver_interface=127.0.98.62
--webserver_port=33531
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38293
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:49:30.248407 14659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:49:30.248849 14659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:49:30.248911 14659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:49:30.253830 14659 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:49:30.253928 14659 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:49:30.253952 14659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:49:30.253969 14659 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:49:30.253985 14659 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:49:30.259918 14659 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38293
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38293
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=33531
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14659
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:49:30.261065 14659 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:49:30.262092 14659 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:49:30.276501 14665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:49:30.276506 14668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:49:30.276501 14666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:49:30.276664 14659 server_base.cc:1034] running on GCE node
I20260619 14:49:30.277639 14659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:49:30.278968 14659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:49:30.280201 14659 hybrid_clock.cc:648] HybridClock initialized: now 1781880570280186 us; error 37 us; skew 500 ppm
Jun 19 14:49:30 dist-test-slave-2sb1 krb5kdc[14111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880570, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:49:30.284246 14659 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 14:49:30.285533 14659 webserver.cc:460] Webserver started at http://127.0.98.62:33531/ using document root <none> and password file <none>
I20260619 14:49:30.286198 14659 fs_manager.cc:359] Metadata directory not provided
I20260619 14:49:30.286324 14659 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:49:30.291023 14659 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:49:30.294029 14675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:30.296132 14659 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 14:49:30.296391 14659 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/master-0/wal
uuid: "b6771a03d5c94a049845292e361e9034"
format_stamp: "Formatted at 2026-06-19 14:49:28 on dist-test-slave-2sb1"
server_key: "9a323fb970da1d2676ccf25ffee35f3d"
server_key_iv: "742b667a3ad03cebfdddd1ad034f72b9"
server_key_version: "encryptionkey@0"
I20260619 14:49:30.297322 14659 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:49:30.311527 14659 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:49:30.315409 14659 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:49:30.315752 14659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:49:30.324098 14659 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38293
I20260619 14:49:30.324433 14727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38293 every 8 connection(s)
I20260619 14:49:30.325377 14659 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:49:30.326593 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 14659
I20260619 14:49:30.327064 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:44465
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:38293
--webserver_port=42965
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:49:30.330443 14728 sys_catalog.cc:263] Verifying existing consensus state
I20260619 14:49:30.332401 14728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6771a03d5c94a049845292e361e9034: Bootstrap starting.
I20260619 14:49:30.350447 14728 log.cc:824] T 00000000000000000000000000000000 P b6771a03d5c94a049845292e361e9034: Log is configured to *not* fsync() on all Append() calls
I20260619 14:49:30.364045 14728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6771a03d5c94a049845292e361e9034: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 14:49:30.364580 14728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6771a03d5c94a049845292e361e9034: Bootstrap complete.
I20260619 14:49:30.369439 14728 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6771a03d5c94a049845292e361e9034 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6771a03d5c94a049845292e361e9034" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38293 } }
I20260619 14:49:30.369882 14728 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6771a03d5c94a049845292e361e9034 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6771a03d5c94a049845292e361e9034, State: Initialized, Role: FOLLOWER
I20260619 14:49:30.370419 14728 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6771a03d5c94a049845292e361e9034 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6771a03d5c94a049845292e361e9034" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38293 } }
I20260619 14:49:30.370540 14728 raft_consensus.cc:388] T 00000000000000000000000000000000 P b6771a03d5c94a049845292e361e9034 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:49:30.370643 14728 raft_consensus.cc:482] T 00000000000000000000000000000000 P b6771a03d5c94a049845292e361e9034 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:49:30.370765 14728 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6771a03d5c94a049845292e361e9034 [term 1 FOLLOWER]: Advancing to term 2
I20260619 14:49:30.373131 14728 raft_consensus.cc:504] T 00000000000000000000000000000000 P b6771a03d5c94a049845292e361e9034 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6771a03d5c94a049845292e361e9034" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38293 } }
I20260619 14:49:30.373538 14728 leader_election.cc:302] T 00000000000000000000000000000000 P b6771a03d5c94a049845292e361e9034 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b6771a03d5c94a049845292e361e9034; no voters:
I20260619 14:49:30.373946 14728 leader_election.cc:288] T 00000000000000000000000000000000 P b6771a03d5c94a049845292e361e9034 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 14:49:30.374002 14732 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b6771a03d5c94a049845292e361e9034 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 14:49:30.374246 14732 raft_consensus.cc:686] T 00000000000000000000000000000000 P b6771a03d5c94a049845292e361e9034 [term 2 LEADER]: Becoming Leader. State: Replica: b6771a03d5c94a049845292e361e9034, State: Running, Role: LEADER
I20260619 14:49:30.375003 14732 consensus_queue.cc:237] T 00000000000000000000000000000000 P b6771a03d5c94a049845292e361e9034 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6771a03d5c94a049845292e361e9034" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38293 } }
I20260619 14:49:30.375928 14728 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6771a03d5c94a049845292e361e9034 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:49:30.376833 14734 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6771a03d5c94a049845292e361e9034 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b6771a03d5c94a049845292e361e9034" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6771a03d5c94a049845292e361e9034" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38293 } } }
I20260619 14:49:30.376981 14734 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6771a03d5c94a049845292e361e9034 [sys.catalog]: This master's current role is: LEADER
I20260619 14:49:30.377401 14733 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6771a03d5c94a049845292e361e9034 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6771a03d5c94a049845292e361e9034. Latest consensus state: current_term: 2 leader_uuid: "b6771a03d5c94a049845292e361e9034" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6771a03d5c94a049845292e361e9034" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38293 } } }
I20260619 14:49:30.377499 14733 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6771a03d5c94a049845292e361e9034 [sys.catalog]: This master's current role is: LEADER
I20260619 14:49:30.378541 14736 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:49:30.384557 14736 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:49:30.385658 14736 catalog_manager.cc:1259] Loaded cluster ID: 3dd5a58639204044aeadf22270a16d68
I20260619 14:49:30.385740 14736 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:49:30.387378 14736 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260619 14:49:30.392480 14736 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260619 14:49:30.399204 14736 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:49:30.400163 14736 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b6771a03d5c94a049845292e361e9034: Loaded TSK: 0
I20260619 14:49:30.401767 14736 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 14:49:30.466513 14730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:49:30.466877 14730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:49:30.466964 14730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:49:30.471393 14730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:49:30.471519 14730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:49:30.471622 14730 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 14:49:30.479736 14730 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:44465
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=42965
--tserver_master_addrs=127.0.98.62:38293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14730
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:49:30.481074 14730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:49:30.482348 14730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:49:30.494475 14755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:49:30.494475 14756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:49:30.494709 14758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:49:30.495174 14730 server_base.cc:1034] running on GCE node
I20260619 14:49:30.495639 14730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:49:30.496434 14730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:49:30.497638 14730 hybrid_clock.cc:648] HybridClock initialized: now 1781880570497601 us; error 66 us; skew 500 ppm
Jun 19 14:49:30 dist-test-slave-2sb1 krb5kdc[14111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880570, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:49:30.501226 14730 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 14:49:30.502456 14730 webserver.cc:460] Webserver started at http://127.0.98.1:42965/ using document root <none> and password file <none>
I20260619 14:49:30.503141 14730 fs_manager.cc:359] Metadata directory not provided
I20260619 14:49:30.503268 14730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:49:30.509258 14730 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 14:49:30.512187 14765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:30.514024 14730 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:49:30.514217 14730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-0/wal
uuid: "756f220b41d0464b947550c9f879f31b"
format_stamp: "Formatted at 2026-06-19 14:49:29 on dist-test-slave-2sb1"
server_key: "d7ec8fd1d6cde41b65d369fc1d9e6ebc"
server_key_iv: "530e145e2ffdec8a5d393ce35f172ff2"
server_key_version: "encryptionkey@0"
I20260619 14:49:30.514842 14730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:49:30.529100 14730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:49:30.533243 14730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:49:30.533644 14730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:49:30.534654 14730 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:49:30.536056 14772 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 14:49:30.542608 14730 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 14:49:30.542743 14730 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.001s sys 0.000s
I20260619 14:49:30.542833 14730 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 14:49:30.545006 14730 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 14:49:30.545099 14730 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260619 14:49:30.545413 14772 tablet_bootstrap.cc:492] T fd0255808a824cafb15d5e1a2b922acf P 756f220b41d0464b947550c9f879f31b: Bootstrap starting.
I20260619 14:49:30.557953 14730 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44465
I20260619 14:49:30.558058 14879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44465 every 8 connection(s)
I20260619 14:49:30.559108 14730 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-0/data/info.pb
I20260619 14:49:30.566764 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 14730
I20260619 14:49:30.567291 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:34133
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:38293
--webserver_port=33907
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:49:30.568406 14772 log.cc:824] T fd0255808a824cafb15d5e1a2b922acf P 756f220b41d0464b947550c9f879f31b: Log is configured to *not* fsync() on all Append() calls
Jun 19 14:49:30 dist-test-slave-2sb1 krb5kdc[14111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880570, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:49:30.578437 14772 tablet_bootstrap.cc:492] T fd0255808a824cafb15d5e1a2b922acf P 756f220b41d0464b947550c9f879f31b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 14:49:30.578967 14772 tablet_bootstrap.cc:492] T fd0255808a824cafb15d5e1a2b922acf P 756f220b41d0464b947550c9f879f31b: Bootstrap complete.
I20260619 14:49:30.580379 14772 ts_tablet_manager.cc:1397] T fd0255808a824cafb15d5e1a2b922acf P 756f220b41d0464b947550c9f879f31b: Time spent bootstrapping tablet: real 0.035s user 0.027s sys 0.004s
I20260619 14:49:30.583630 14882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:30.561827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59693 (local address 127.0.98.62:38293)
0619 14:49:30.562366 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:30.562470 (+ 104us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:30.562646 (+ 176us) server_negotiation.cc:407] Connection header received
0619 14:49:30.564030 (+ 1384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:30.564047 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:30.564937 (+ 890us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:30.565547 (+ 610us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:49:30.567563 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:30.569271 (+ 1708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:30.570746 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:30.571481 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:30.571678 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:30.575815 (+ 4137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:49:30.575824 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:49:30.575841 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:49:30.575882 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:49:30.578562 (+ 2680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:30.579355 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:30.579543 (+ 188us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:30.579556 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:30.579762 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:30.580353 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:30.580356 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:30.580358 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:30.581393 (+ 1035us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:49:30.581573 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:30.582367 (+ 794us) server_negotiation.cc:299] Negotiation successful
0619 14:49:30.582694 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":168,"threads_started":1}
I20260619 14:49:30.583664 14883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:30.561800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38293 (local address 127.0.98.1:59693)
0619 14:49:30.562448 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:30.562507 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 14:49:30.563471 (+ 964us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:49:30.565724 (+ 2253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:49:30.565732 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:49:30.566501 (+ 769us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:49:30.567295 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:30.567321 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:30.569480 (+ 2159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:30.569487 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:30.570573 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:30.570589 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:30.571683 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:30.571688 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:30.571932 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:30.572741 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:49:30.572770 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:49:30.575463 (+ 2693us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:49:30.578759 (+ 3296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:49:30.578769 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:49:30.578786 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:49:30.579202 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:49:30.580023 (+ 821us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:49:30.580030 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:49:30.580034 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:49:30.580229 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:49:30.581597 (+ 1368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:49:30.581605 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:49:30.581960 (+ 355us) client_negotiation.cc:769] Sending connection context
0619 14:49:30.582210 (+ 250us) client_negotiation.cc:240] Negotiation successful
0619 14:49:30.582619 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":144,"threads_started":1}
I20260619 14:49:30.584190 14772 raft_consensus.cc:348] T fd0255808a824cafb15d5e1a2b922acf P 756f220b41d0464b947550c9f879f31b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "756f220b41d0464b947550c9f879f31b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44465 } } peers { permanent_uuid: "0cf4adf2a6664d31af185aeb05846b9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33209 } } peers { permanent_uuid: "7f8375b6be764013974e4e579f7a6e80" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34133 } }
I20260619 14:49:30.584872 14772 raft_consensus.cc:729] T fd0255808a824cafb15d5e1a2b922acf P 756f220b41d0464b947550c9f879f31b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 756f220b41d0464b947550c9f879f31b, State: Initialized, Role: FOLLOWER
I20260619 14:49:30.585433 14772 consensus_queue.cc:260] T fd0255808a824cafb15d5e1a2b922acf P 756f220b41d0464b947550c9f879f31b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "756f220b41d0464b947550c9f879f31b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44465 } } peers { permanent_uuid: "0cf4adf2a6664d31af185aeb05846b9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33209 } } peers { permanent_uuid: "7f8375b6be764013974e4e579f7a6e80" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34133 } }
I20260619 14:49:30.586570 14772 ts_tablet_manager.cc:1428] T fd0255808a824cafb15d5e1a2b922acf P 756f220b41d0464b947550c9f879f31b: Time spent starting tablet: real 0.006s user 0.005s sys 0.004s
I20260619 14:49:30.586993 14880 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38293
I20260619 14:49:30.587255 14880 heartbeater.cc:464] Registering TS with master...
I20260619 14:49:30.587846 14880 heartbeater.cc:511] Master 127.0.98.62:38293 requested a full tablet report, sending...
I20260619 14:49:30.591010 14687 ts_manager.cc:194] Registered new tserver with Master: 756f220b41d0464b947550c9f879f31b (127.0.98.1:44465)
I20260619 14:49:30.595094 14687 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:59693
I20260619 14:49:30.598109 14880 heartbeater.cc:503] Master 127.0.98.62:38293 was elected leader, sending a full tablet report...
Jun 19 14:49:30 dist-test-slave-2sb1 krb5kdc[14111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880570, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 14:49:30.607671 14888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:30.594736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38152 (local address 127.0.98.1:44465)
0619 14:49:30.595510 (+ 774us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:30.595522 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:30.595551 (+ 29us) server_negotiation.cc:407] Connection header received
0619 14:49:30.595747 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:30.595752 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:30.595976 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:30.596123 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:49:30.596900 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:30.597635 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:30.598716 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:30.599226 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:30.599345 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:30.602435 (+ 3090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:49:30.602454 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:49:30.602466 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:49:30.602495 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:49:30.604809 (+ 2314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:30.605670 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:30.605677 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:30.605678 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:30.605795 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:30.606264 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:30.606268 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:30.606269 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:30.606733 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:49:30.606894 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:30.607364 (+ 470us) server_negotiation.cc:299] Negotiation successful
0619 14:49:30.607530 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":668,"thread_start_us":128,"threads_started":1}
I20260619 14:49:30.607720 14887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:30.594578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44465 (local address 127.0.0.1:38152)
0619 14:49:30.595368 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:30.595430 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0619 14:49:30.595575 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:49:30.596144 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:49:30.596150 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:49:30.596554 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:49:30.596753 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:30.596762 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:30.597770 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:30.597776 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:30.598562 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:30.598575 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:30.599361 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:30.599379 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:30.599524 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:30.600259 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:49:30.600273 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:49:30.602120 (+ 1847us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:49:30.604992 (+ 2872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:49:30.605009 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:49:30.605015 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:49:30.605419 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:49:30.605913 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:49:30.605916 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:49:30.605919 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:49:30.606036 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:49:30.606935 (+ 899us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:49:30.606942 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:49:30.607098 (+ 156us) client_negotiation.cc:769] Sending connection context
0619 14:49:30.607268 (+ 170us) client_negotiation.cc:240] Negotiation successful
0619 14:49:30.607521 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":121,"threads_started":1}
I20260619 14:49:30.609050 14813 tablet_service.cc:1514] Processing DeleteTablet for tablet fd0255808a824cafb15d5e1a2b922acf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:49:30 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38152
I20260619 14:49:30.610020 14889 tablet_replica.cc:330] stopping tablet replica
I20260619 14:49:30.610399 14889 raft_consensus.cc:2229] T fd0255808a824cafb15d5e1a2b922acf P 756f220b41d0464b947550c9f879f31b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:49:30.610715 14889 raft_consensus.cc:2258] T fd0255808a824cafb15d5e1a2b922acf P 756f220b41d0464b947550c9f879f31b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:49:30.612115 14889 ts_tablet_manager.cc:1905] T fd0255808a824cafb15d5e1a2b922acf P 756f220b41d0464b947550c9f879f31b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:49:30.618649 14889 ts_tablet_manager.cc:1918] T fd0255808a824cafb15d5e1a2b922acf P 756f220b41d0464b947550c9f879f31b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:49:30.618825 14889 log.cc:1196] T fd0255808a824cafb15d5e1a2b922acf P 756f220b41d0464b947550c9f879f31b: Deleting WAL directory at /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-0/wal/wals/fd0255808a824cafb15d5e1a2b922acf
I20260619 14:49:30.619158 14889 ts_tablet_manager.cc:1939] T fd0255808a824cafb15d5e1a2b922acf P 756f220b41d0464b947550c9f879f31b: Deleting consensus metadata
I20260619 14:49:30.621738 14679 catalog_manager.cc:4841] TS 756f220b41d0464b947550c9f879f31b (127.0.98.1:44465): tablet fd0255808a824cafb15d5e1a2b922acf (table test-table [id=13d295a8621d48e59bc208f1dc079096]) successfully deleted
W20260619 14:49:30.707765 14884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:49:30.708081 14884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:49:30.708127 14884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:49:30.712379 14884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:49:30.712466 14884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:49:30.712529 14884 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 14:49:30.719017 14884 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:34133
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=33907
--tserver_master_addrs=127.0.98.62:38293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14884
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:49:30.720288 14884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:49:30.721410 14884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:49:30.732147 14897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:49:30.732198 14894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:49:30.732147 14895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:49:30.732823 14884 server_base.cc:1034] running on GCE node
I20260619 14:49:30.733402 14884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:49:30.734115 14884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:49:30.737507 14884 hybrid_clock.cc:648] HybridClock initialized: now 1781880570737473 us; error 51 us; skew 500 ppm
Jun 19 14:49:30 dist-test-slave-2sb1 krb5kdc[14111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880570, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:49:30.741250 14884 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 14:49:30.742669 14884 webserver.cc:460] Webserver started at http://127.0.98.2:33907/ using document root <none> and password file <none>
I20260619 14:49:30.743337 14884 fs_manager.cc:359] Metadata directory not provided
I20260619 14:49:30.743474 14884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:49:30.747982 14884 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.001s
I20260619 14:49:30.750771 14904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:30.752684 14884 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 14:49:30.752878 14884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-1/wal
uuid: "7f8375b6be764013974e4e579f7a6e80"
format_stamp: "Formatted at 2026-06-19 14:49:29 on dist-test-slave-2sb1"
server_key: "75e11f0fa2159637d278bbc0f234423a"
server_key_iv: "dc2c124dcddeaee51cdca4d7b066e3dd"
server_key_version: "encryptionkey@0"
I20260619 14:49:30.753504 14884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:49:30.788385 14884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:49:30.792614 14884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:49:30.792924 14884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:49:30.793628 14884 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:49:30.794531 14884 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:49:30.794611 14884 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:30.794687 14884 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:49:30.794734 14884 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:30.806463 14884 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34133
I20260619 14:49:30.806550 15017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34133 every 8 connection(s)
I20260619 14:49:30.807658 14884 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-1/data/info.pb
I20260619 14:49:30.816521 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 14884
I20260619 14:49:30.817039 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:33209
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:38293
--webserver_port=44395
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 14:49:30 dist-test-slave-2sb1 krb5kdc[14111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880570, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:49:30.825588 14882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:30.810043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59403 (local address 127.0.98.62:38293)
0619 14:49:30.810204 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:30.810208 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:30.811902 (+ 1694us) server_negotiation.cc:407] Connection header received
0619 14:49:30.812829 (+ 927us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:30.812832 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:30.812908 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:30.813016 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:49:30.814323 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:30.814865 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:30.816113 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:30.816356 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:30.816473 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:30.819593 (+ 3120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:49:30.819599 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:49:30.819602 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:49:30.819638 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:49:30.821800 (+ 2162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:30.822559 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:30.822564 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:30.822566 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:30.822663 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:30.823147 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:30.823151 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:30.823153 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:30.823825 (+ 672us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:49:30.824085 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:30.825193 (+ 1108us) server_negotiation.cc:299] Negotiation successful
0619 14:49:30.825383 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260619 14:49:30.826546 15020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:30.810525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38293 (local address 127.0.98.2:59403)
0619 14:49:30.811582 (+ 1057us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:30.811631 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0619 14:49:30.812541 (+ 910us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:49:30.813203 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:49:30.813210 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:49:30.813664 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:49:30.814150 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:30.814164 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:30.814996 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:30.815005 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:30.815965 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:30.815976 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:30.816520 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:30.816524 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:30.816779 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:30.817483 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:49:30.817501 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:49:30.819341 (+ 1840us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:49:30.821996 (+ 2655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:49:30.822005 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:49:30.822034 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:49:30.822418 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:49:30.822778 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:49:30.822782 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:49:30.822784 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:49:30.823027 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:49:30.824273 (+ 1246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:49:30.824307 (+ 34us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:49:30.824841 (+ 534us) client_negotiation.cc:769] Sending connection context
0619 14:49:30.825116 (+ 275us) client_negotiation.cc:240] Negotiation successful
0619 14:49:30.825553 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":845,"thread_start_us":645,"threads_started":1}
I20260619 14:49:30.828274 15018 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38293
I20260619 14:49:30.828727 15018 heartbeater.cc:464] Registering TS with master...
I20260619 14:49:30.829380 15018 heartbeater.cc:511] Master 127.0.98.62:38293 requested a full tablet report, sending...
I20260619 14:49:30.830741 14687 ts_manager.cc:194] Registered new tserver with Master: 7f8375b6be764013974e4e579f7a6e80 (127.0.98.2:34133)
I20260619 14:49:30.831578 14687 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:59403
W20260619 14:49:30.953017 15021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:49:30.953711 15021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:49:30.953999 15021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:49:30.959169 15021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:49:30.959288 15021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:49:30.959393 15021 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 14:49:30.965893 15021 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:33209
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=44395
--tserver_master_addrs=127.0.98.62:38293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15021
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:49:30.967187 15021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:49:30.968367 15021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:49:30.981037 15029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:49:30.981016 15027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:49:30.981016 15026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:49:30.981104 15021 server_base.cc:1034] running on GCE node
I20260619 14:49:30.981633 15021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:49:30.982286 15021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:49:30.983485 15021 hybrid_clock.cc:648] HybridClock initialized: now 1781880570983468 us; error 29 us; skew 500 ppm
Jun 19 14:49:30 dist-test-slave-2sb1 krb5kdc[14111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880570, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:49:30.987010 15021 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 14:49:30.988096 15021 webserver.cc:460] Webserver started at http://127.0.98.3:44395/ using document root <none> and password file <none>
I20260619 14:49:30.988796 15021 fs_manager.cc:359] Metadata directory not provided
I20260619 14:49:30.988912 15021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:49:30.993728 15021 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:49:30.996335 15036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:30.997992 15021 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:49:30.998169 15021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-2/wal
uuid: "0cf4adf2a6664d31af185aeb05846b9d"
format_stamp: "Formatted at 2026-06-19 14:49:29 on dist-test-slave-2sb1"
server_key: "67bcf70d36ed12b1ba5463a5836affec"
server_key_iv: "057d4659b4200470957a6190fdac9a5d"
server_key_version: "encryptionkey@0"
I20260619 14:49:30.998828 15021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:49:31.033915 15021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:49:31.037978 15021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:49:31.038232 15021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:49:31.038972 15021 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:49:31.040026 15021 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:49:31.040107 15021 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:31.040187 15021 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:49:31.040246 15021 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:31.051844 15021 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33209
I20260619 14:49:31.051963 15149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33209 every 8 connection(s)
I20260619 14:49:31.053226 15021 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-2/data/info.pb
I20260619 14:49:31.056732 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 15021
Jun 19 14:49:31 dist-test-slave-2sb1 krb5kdc[14111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880570, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:49:31.069033 15153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:31.058087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37084 (local address 127.0.98.62:38293)
0619 14:49:31.058578 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:31.058582 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:31.058598 (+ 16us) server_negotiation.cc:407] Connection header received
0619 14:49:31.058671 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:31.058674 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:31.058745 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:31.060163 (+ 1418us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:49:31.060301 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:31.061112 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:31.062545 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:31.062922 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:31.063063 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:31.064787 (+ 1724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:49:31.064791 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:49:31.064794 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:49:31.064822 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:49:31.066845 (+ 2023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:31.067320 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:31.067323 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:31.067325 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:31.067573 (+ 248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:31.067968 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:31.067972 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:31.067974 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:31.068224 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:49:31.068450 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:31.068581 (+ 131us) server_negotiation.cc:299] Negotiation successful
0619 14:49:31.068786 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":69,"threads_started":1}
I20260619 14:49:31.074625 14882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:31.058004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47635 (local address 127.0.98.62:38293)
0619 14:49:31.058275 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:31.058280 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:31.059033 (+ 753us) server_negotiation.cc:407] Connection header received
0619 14:49:31.061630 (+ 2597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:31.061633 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:31.061687 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:31.061766 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:49:31.063520 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:31.064328 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:31.066106 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:31.066490 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:31.066623 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:31.069697 (+ 3074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:49:31.069700 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:49:31.069702 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:49:31.069745 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:49:31.071394 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:31.072449 (+ 1055us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:31.072452 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:31.072454 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:31.072541 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:31.073130 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:31.073133 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:31.073135 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:31.073410 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:49:31.073528 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:31.074259 (+ 731us) server_negotiation.cc:299] Negotiation successful
0619 14:49:31.074478 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189}
I20260619 14:49:31.075639 15154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:31.058189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38293 (local address 127.0.98.3:47635)
0619 14:49:31.058855 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:31.058921 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0619 14:49:31.061296 (+ 2375us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:49:31.062104 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:49:31.062137 (+ 33us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:49:31.062737 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:49:31.063358 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:31.063372 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:31.064490 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:31.064493 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:31.065946 (+ 1453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:31.065964 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:31.066673 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:31.066676 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:31.066970 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:31.067587 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:49:31.067605 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:49:31.069496 (+ 1891us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:49:31.071547 (+ 2051us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:49:31.071553 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:49:31.071567 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:49:31.072235 (+ 668us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:49:31.072764 (+ 529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:49:31.072768 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:49:31.072770 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:49:31.072979 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:49:31.073549 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:49:31.073555 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:49:31.073854 (+ 299us) client_negotiation.cc:769] Sending connection context
0619 14:49:31.074140 (+ 286us) client_negotiation.cc:240] Negotiation successful
0619 14:49:31.074549 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":184,"threads_started":1}
I20260619 14:49:31.077015 15150 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38293
I20260619 14:49:31.077605 15150 heartbeater.cc:464] Registering TS with master...
I20260619 14:49:31.078286 15150 heartbeater.cc:511] Master 127.0.98.62:38293 requested a full tablet report, sending...
I20260619 14:49:31.079937 14687 ts_manager.cc:194] Registered new tserver with Master: 0cf4adf2a6664d31af185aeb05846b9d (127.0.98.3:33209)
I20260619 14:49:31.080569 14687 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47635
I20260619 14:49:31.084048 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:49:31.089736 14882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:31.084795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37100 (local address 127.0.98.62:38293)
0619 14:49:31.085056 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:31.085060 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:31.085096 (+ 36us) server_negotiation.cc:407] Connection header received
0619 14:49:31.085310 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:31.085314 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:31.085457 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:31.085553 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:49:31.085975 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:31.086671 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:31.087638 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:31.087899 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:31.088001 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:31.088413 (+ 412us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:49:31.089195 (+ 782us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:49:31.089299 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:31.089536 (+ 237us) server_negotiation.cc:299] Negotiation successful
0619 14:49:31.089607 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 14:49:31.090744 14687 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:37100:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:49:31.094622 14687 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:49:31.104964 14813 tablet_service.cc:1467] Processing CreateTablet for tablet 6e599e3008f64ced9dcd0601e19e64b9 (DEFAULT_TABLE table=test-table [id=9a74b6a91b39418a895b2ce120ae4396]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:49:31.106066 14813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e599e3008f64ced9dcd0601e19e64b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:49:31.109922 15162 tablet_bootstrap.cc:492] T 6e599e3008f64ced9dcd0601e19e64b9 P 756f220b41d0464b947550c9f879f31b: Bootstrap starting.
I20260619 14:49:31.110281 14887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:31.103737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33209 (local address 127.0.0.1:44590)
0619 14:49:31.104163 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:31.104180 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 14:49:31.104268 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:49:31.105266 (+ 998us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:49:31.105269 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:49:31.105290 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:49:31.105425 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:31.105432 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:31.107226 (+ 1794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:31.107230 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:31.108788 (+ 1558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:31.108802 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:31.109744 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:31.109751 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:31.109873 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:31.109936 (+ 63us) client_negotiation.cc:769] Sending connection context
0619 14:49:31.110010 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 14:49:31.110117 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"mutex_wait_us":39}
I20260619 14:49:31.111029 15162 tablet_bootstrap.cc:654] T 6e599e3008f64ced9dcd0601e19e64b9 P 756f220b41d0464b947550c9f879f31b: Neither blocks nor log segments found. Creating new log.
I20260619 14:49:31.111204 15161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:31.104446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44590 (local address 127.0.98.3:33209)
0619 14:49:31.104780 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:31.104794 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:31.104824 (+ 30us) server_negotiation.cc:407] Connection header received
0619 14:49:31.104894 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:31.104899 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:31.105104 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:31.105349 (+ 245us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:49:31.105567 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:31.107042 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:31.108939 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:31.109621 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:31.110285 (+ 664us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:31.110752 (+ 467us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:31.110815 (+ 63us) server_negotiation.cc:299] Negotiation successful
0619 14:49:31.110940 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":102,"threads_started":1}
I20260619 14:49:31.111225 15159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:31.104029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34133 (local address 127.0.0.1:58554)
0619 14:49:31.104372 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:31.104386 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:49:31.104461 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:49:31.106308 (+ 1847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:49:31.106311 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:49:31.106330 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:49:31.106423 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:31.106428 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:31.107764 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:31.107768 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:31.108835 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:31.108848 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:31.110786 (+ 1938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:31.110790 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:31.110888 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:31.110913 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 14:49:31.110983 (+ 70us) client_negotiation.cc:240] Negotiation successful
0619 14:49:31.111076 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"mutex_wait_us":5,"thread_start_us":122,"threads_started":1}
I20260619 14:49:31.111358 15160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:31.104234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58554 (local address 127.0.98.2:34133)
0619 14:49:31.105755 (+ 1521us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:31.105767 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:31.105793 (+ 26us) server_negotiation.cc:407] Connection header received
0619 14:49:31.105874 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:31.105882 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:31.106146 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:31.106573 (+ 427us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:49:31.106735 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:31.107622 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:31.109835 (+ 2213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:31.110643 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:31.110848 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:31.111050 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:31.111106 (+ 56us) server_negotiation.cc:299] Negotiation successful
0619 14:49:31.111206 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1404,"thread_start_us":151,"threads_started":1}
I20260619 14:49:31.112520 15162 tablet_bootstrap.cc:492] T 6e599e3008f64ced9dcd0601e19e64b9 P 756f220b41d0464b947550c9f879f31b: No bootstrap required, opened a new log
I20260619 14:49:31.112646 15162 ts_tablet_manager.cc:1397] T 6e599e3008f64ced9dcd0601e19e64b9 P 756f220b41d0464b947550c9f879f31b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260619 14:49:31.113188 15162 raft_consensus.cc:348] T 6e599e3008f64ced9dcd0601e19e64b9 P 756f220b41d0464b947550c9f879f31b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf4adf2a6664d31af185aeb05846b9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33209 } } peers { permanent_uuid: "7f8375b6be764013974e4e579f7a6e80" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34133 } } peers { permanent_uuid: "756f220b41d0464b947550c9f879f31b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44465 } }
I20260619 14:49:31.113334 15162 raft_consensus.cc:374] T 6e599e3008f64ced9dcd0601e19e64b9 P 756f220b41d0464b947550c9f879f31b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:49:31.113384 15162 raft_consensus.cc:729] T 6e599e3008f64ced9dcd0601e19e64b9 P 756f220b41d0464b947550c9f879f31b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 756f220b41d0464b947550c9f879f31b, State: Initialized, Role: FOLLOWER
I20260619 14:49:31.113544 15162 consensus_queue.cc:260] T 6e599e3008f64ced9dcd0601e19e64b9 P 756f220b41d0464b947550c9f879f31b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf4adf2a6664d31af185aeb05846b9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33209 } } peers { permanent_uuid: "7f8375b6be764013974e4e579f7a6e80" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34133 } } peers { permanent_uuid: "756f220b41d0464b947550c9f879f31b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44465 } }
I20260619 14:49:31.113997 15162 ts_tablet_manager.cc:1428] T 6e599e3008f64ced9dcd0601e19e64b9 P 756f220b41d0464b947550c9f879f31b: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260619 14:49:31.114861 15084 tablet_service.cc:1467] Processing CreateTablet for tablet 6e599e3008f64ced9dcd0601e19e64b9 (DEFAULT_TABLE table=test-table [id=9a74b6a91b39418a895b2ce120ae4396]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:49:31.114861 14952 tablet_service.cc:1467] Processing CreateTablet for tablet 6e599e3008f64ced9dcd0601e19e64b9 (DEFAULT_TABLE table=test-table [id=9a74b6a91b39418a895b2ce120ae4396]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:49:31.116263 15084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e599e3008f64ced9dcd0601e19e64b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:49:31.116250 14952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e599e3008f64ced9dcd0601e19e64b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:49:31.125886 15166 tablet_bootstrap.cc:492] T 6e599e3008f64ced9dcd0601e19e64b9 P 7f8375b6be764013974e4e579f7a6e80: Bootstrap starting.
I20260619 14:49:31.126459 15165 tablet_bootstrap.cc:492] T 6e599e3008f64ced9dcd0601e19e64b9 P 0cf4adf2a6664d31af185aeb05846b9d: Bootstrap starting.
I20260619 14:49:31.128877 15165 tablet_bootstrap.cc:654] T 6e599e3008f64ced9dcd0601e19e64b9 P 0cf4adf2a6664d31af185aeb05846b9d: Neither blocks nor log segments found. Creating new log.
I20260619 14:49:31.129485 15166 tablet_bootstrap.cc:654] T 6e599e3008f64ced9dcd0601e19e64b9 P 7f8375b6be764013974e4e579f7a6e80: Neither blocks nor log segments found. Creating new log.
I20260619 14:49:31.130450 15166 log.cc:824] T 6e599e3008f64ced9dcd0601e19e64b9 P 7f8375b6be764013974e4e579f7a6e80: Log is configured to *not* fsync() on all Append() calls
I20260619 14:49:31.130450 15165 log.cc:824] T 6e599e3008f64ced9dcd0601e19e64b9 P 0cf4adf2a6664d31af185aeb05846b9d: Log is configured to *not* fsync() on all Append() calls
I20260619 14:49:31.133210 15166 tablet_bootstrap.cc:492] T 6e599e3008f64ced9dcd0601e19e64b9 P 7f8375b6be764013974e4e579f7a6e80: No bootstrap required, opened a new log
I20260619 14:49:31.133198 15165 tablet_bootstrap.cc:492] T 6e599e3008f64ced9dcd0601e19e64b9 P 0cf4adf2a6664d31af185aeb05846b9d: No bootstrap required, opened a new log
I20260619 14:49:31.133435 15165 ts_tablet_manager.cc:1397] T 6e599e3008f64ced9dcd0601e19e64b9 P 0cf4adf2a6664d31af185aeb05846b9d: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.004s
I20260619 14:49:31.133435 15166 ts_tablet_manager.cc:1397] T 6e599e3008f64ced9dcd0601e19e64b9 P 7f8375b6be764013974e4e579f7a6e80: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260619 14:49:31.138309 15165 raft_consensus.cc:348] T 6e599e3008f64ced9dcd0601e19e64b9 P 0cf4adf2a6664d31af185aeb05846b9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf4adf2a6664d31af185aeb05846b9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33209 } } peers { permanent_uuid: "7f8375b6be764013974e4e579f7a6e80" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34133 } } peers { permanent_uuid: "756f220b41d0464b947550c9f879f31b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44465 } }
I20260619 14:49:31.138309 15166 raft_consensus.cc:348] T 6e599e3008f64ced9dcd0601e19e64b9 P 7f8375b6be764013974e4e579f7a6e80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf4adf2a6664d31af185aeb05846b9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33209 } } peers { permanent_uuid: "7f8375b6be764013974e4e579f7a6e80" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34133 } } peers { permanent_uuid: "756f220b41d0464b947550c9f879f31b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44465 } }
I20260619 14:49:31.138672 15165 raft_consensus.cc:374] T 6e599e3008f64ced9dcd0601e19e64b9 P 0cf4adf2a6664d31af185aeb05846b9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:49:31.138672 15166 raft_consensus.cc:374] T 6e599e3008f64ced9dcd0601e19e64b9 P 7f8375b6be764013974e4e579f7a6e80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:49:31.138748 15165 raft_consensus.cc:729] T 6e599e3008f64ced9dcd0601e19e64b9 P 0cf4adf2a6664d31af185aeb05846b9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cf4adf2a6664d31af185aeb05846b9d, State: Initialized, Role: FOLLOWER
I20260619 14:49:31.138758 15166 raft_consensus.cc:729] T 6e599e3008f64ced9dcd0601e19e64b9 P 7f8375b6be764013974e4e579f7a6e80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f8375b6be764013974e4e579f7a6e80, State: Initialized, Role: FOLLOWER
I20260619 14:49:31.139289 15166 consensus_queue.cc:260] T 6e599e3008f64ced9dcd0601e19e64b9 P 7f8375b6be764013974e4e579f7a6e80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf4adf2a6664d31af185aeb05846b9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33209 } } peers { permanent_uuid: "7f8375b6be764013974e4e579f7a6e80" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34133 } } peers { permanent_uuid: "756f220b41d0464b947550c9f879f31b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44465 } }
I20260619 14:49:31.139509 15165 consensus_queue.cc:260] T 6e599e3008f64ced9dcd0601e19e64b9 P 0cf4adf2a6664d31af185aeb05846b9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf4adf2a6664d31af185aeb05846b9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33209 } } peers { permanent_uuid: "7f8375b6be764013974e4e579f7a6e80" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34133 } } peers { permanent_uuid: "756f220b41d0464b947550c9f879f31b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44465 } }
I20260619 14:49:31.140306 15018 heartbeater.cc:503] Master 127.0.98.62:38293 was elected leader, sending a full tablet report...
I20260619 14:49:31.140580 15150 heartbeater.cc:503] Master 127.0.98.62:38293 was elected leader, sending a full tablet report...
I20260619 14:49:31.140708 15165 ts_tablet_manager.cc:1428] T 6e599e3008f64ced9dcd0601e19e64b9 P 0cf4adf2a6664d31af185aeb05846b9d: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260619 14:49:31.141737 15166 ts_tablet_manager.cc:1428] T 6e599e3008f64ced9dcd0601e19e64b9 P 7f8375b6be764013974e4e579f7a6e80: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260619 14:49:31.247421 15170 raft_consensus.cc:482] T 6e599e3008f64ced9dcd0601e19e64b9 P 7f8375b6be764013974e4e579f7a6e80 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:49:31.247745 15170 raft_consensus.cc:504] T 6e599e3008f64ced9dcd0601e19e64b9 P 7f8375b6be764013974e4e579f7a6e80 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf4adf2a6664d31af185aeb05846b9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33209 } } peers { permanent_uuid: "7f8375b6be764013974e4e579f7a6e80" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34133 } } peers { permanent_uuid: "756f220b41d0464b947550c9f879f31b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44465 } }
I20260619 14:49:31.249174 15170 leader_election.cc:288] T 6e599e3008f64ced9dcd0601e19e64b9 P 7f8375b6be764013974e4e579f7a6e80 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cf4adf2a6664d31af185aeb05846b9d (127.0.98.3:33209), 756f220b41d0464b947550c9f879f31b (127.0.98.1:44465)
I20260619 14:49:31.255309 15161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:31.249761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60411 (local address 127.0.98.3:33209)
0619 14:49:31.249898 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:31.249902 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:31.250142 (+ 240us) server_negotiation.cc:407] Connection header received
0619 14:49:31.250199 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:31.250202 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:31.250247 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:31.250357 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:49:31.250724 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:31.251602 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:31.253917 (+ 2315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:31.254541 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:31.254644 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:31.254756 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:31.255087 (+ 331us) server_negotiation.cc:299] Negotiation successful
0619 14:49:31.255171 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260619 14:49:31.255347 15172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:31.249757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53685 (local address 127.0.98.1:44465)
0619 14:49:31.250077 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:31.250083 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:31.250128 (+ 45us) server_negotiation.cc:407] Connection header received
0619 14:49:31.250192 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:31.250195 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:31.250267 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:31.250378 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:49:31.250724 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:31.251604 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:31.253755 (+ 2151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:31.254498 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:31.254642 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:31.254826 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:31.255087 (+ 261us) server_negotiation.cc:299] Negotiation successful
0619 14:49:31.255173 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":107,"threads_started":1}
I20260619 14:49:31.255414 15020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:31.249530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44465 (local address 127.0.98.2:53685)
0619 14:49:31.249752 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:31.249769 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 14:49:31.249868 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:49:31.250400 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:49:31.250402 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:49:31.250418 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:49:31.250536 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:31.250546 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:31.251920 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:31.251924 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:31.253549 (+ 1625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:31.253564 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:31.254698 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:31.254703 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:31.254901 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:31.254952 (+ 51us) client_negotiation.cc:769] Sending connection context
0619 14:49:31.255047 (+ 95us) client_negotiation.cc:240] Negotiation successful
0619 14:49:31.255161 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":215}
I20260619 14:49:31.255424 15171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:31.249545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33209 (local address 127.0.98.2:60411)
0619 14:49:31.250046 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:31.250058 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:49:31.250138 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:49:31.250400 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:49:31.250468 (+ 68us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:49:31.250481 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:49:31.250571 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:31.250579 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:31.251789 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:31.251793 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:31.253548 (+ 1755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:31.253564 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:31.254698 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:31.254703 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:31.254864 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:31.254944 (+ 80us) client_negotiation.cc:769] Sending connection context
0619 14:49:31.255047 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 14:49:31.255161 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"mutex_wait_us":222,"thread_start_us":137,"threads_started":1}
I20260619 14:49:31.256209 14834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e599e3008f64ced9dcd0601e19e64b9" candidate_uuid: "7f8375b6be764013974e4e579f7a6e80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "756f220b41d0464b947550c9f879f31b" is_pre_election: true
I20260619 14:49:31.256636 15104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e599e3008f64ced9dcd0601e19e64b9" candidate_uuid: "7f8375b6be764013974e4e579f7a6e80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cf4adf2a6664d31af185aeb05846b9d" is_pre_election: true
I20260619 14:49:31.256951 14834 raft_consensus.cc:2454] T 6e599e3008f64ced9dcd0601e19e64b9 P 756f220b41d0464b947550c9f879f31b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f8375b6be764013974e4e579f7a6e80 in term 0.
I20260619 14:49:31.257079 15104 raft_consensus.cc:2454] T 6e599e3008f64ced9dcd0601e19e64b9 P 0cf4adf2a6664d31af185aeb05846b9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f8375b6be764013974e4e579f7a6e80 in term 0.
I20260619 14:49:31.257652 14908 leader_election.cc:302] T 6e599e3008f64ced9dcd0601e19e64b9 P 7f8375b6be764013974e4e579f7a6e80 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 756f220b41d0464b947550c9f879f31b, 7f8375b6be764013974e4e579f7a6e80; no voters:
I20260619 14:49:31.258028 15170 raft_consensus.cc:2781] T 6e599e3008f64ced9dcd0601e19e64b9 P 7f8375b6be764013974e4e579f7a6e80 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:49:31.258118 15170 raft_consensus.cc:482] T 6e599e3008f64ced9dcd0601e19e64b9 P 7f8375b6be764013974e4e579f7a6e80 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:49:31.258152 15170 raft_consensus.cc:3037] T 6e599e3008f64ced9dcd0601e19e64b9 P 7f8375b6be764013974e4e579f7a6e80 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:49:31.259310 15170 raft_consensus.cc:504] T 6e599e3008f64ced9dcd0601e19e64b9 P 7f8375b6be764013974e4e579f7a6e80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf4adf2a6664d31af185aeb05846b9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33209 } } peers { permanent_uuid: "7f8375b6be764013974e4e579f7a6e80" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34133 } } peers { permanent_uuid: "756f220b41d0464b947550c9f879f31b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44465 } }
I20260619 14:49:31.259754 15170 leader_election.cc:288] T 6e599e3008f64ced9dcd0601e19e64b9 P 7f8375b6be764013974e4e579f7a6e80 [CANDIDATE]: Term 1 election: Requested vote from peers 0cf4adf2a6664d31af185aeb05846b9d (127.0.98.3:33209), 756f220b41d0464b947550c9f879f31b (127.0.98.1:44465)
I20260619 14:49:31.260304 15104 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e599e3008f64ced9dcd0601e19e64b9" candidate_uuid: "7f8375b6be764013974e4e579f7a6e80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cf4adf2a6664d31af185aeb05846b9d"
I20260619 14:49:31.260417 14834 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e599e3008f64ced9dcd0601e19e64b9" candidate_uuid: "7f8375b6be764013974e4e579f7a6e80" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "756f220b41d0464b947550c9f879f31b"
I20260619 14:49:31.260468 15104 raft_consensus.cc:3037] T 6e599e3008f64ced9dcd0601e19e64b9 P 0cf4adf2a6664d31af185aeb05846b9d [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:49:31.260588 14834 raft_consensus.cc:3037] T 6e599e3008f64ced9dcd0601e19e64b9 P 756f220b41d0464b947550c9f879f31b [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:49:31.261953 14834 raft_consensus.cc:2454] T 6e599e3008f64ced9dcd0601e19e64b9 P 756f220b41d0464b947550c9f879f31b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f8375b6be764013974e4e579f7a6e80 in term 1.
I20260619 14:49:31.262065 15104 raft_consensus.cc:2454] T 6e599e3008f64ced9dcd0601e19e64b9 P 0cf4adf2a6664d31af185aeb05846b9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f8375b6be764013974e4e579f7a6e80 in term 1.
I20260619 14:49:31.262329 14908 leader_election.cc:302] T 6e599e3008f64ced9dcd0601e19e64b9 P 7f8375b6be764013974e4e579f7a6e80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 756f220b41d0464b947550c9f879f31b, 7f8375b6be764013974e4e579f7a6e80; no voters:
I20260619 14:49:31.262529 15170 raft_consensus.cc:2781] T 6e599e3008f64ced9dcd0601e19e64b9 P 7f8375b6be764013974e4e579f7a6e80 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:49:31.262737 15170 raft_consensus.cc:686] T 6e599e3008f64ced9dcd0601e19e64b9 P 7f8375b6be764013974e4e579f7a6e80 [term 1 LEADER]: Becoming Leader. State: Replica: 7f8375b6be764013974e4e579f7a6e80, State: Running, Role: LEADER
I20260619 14:49:31.263229 15170 consensus_queue.cc:237] T 6e599e3008f64ced9dcd0601e19e64b9 P 7f8375b6be764013974e4e579f7a6e80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf4adf2a6664d31af185aeb05846b9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33209 } } peers { permanent_uuid: "7f8375b6be764013974e4e579f7a6e80" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34133 } } peers { permanent_uuid: "756f220b41d0464b947550c9f879f31b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44465 } }
I20260619 14:49:31.266346 14687 catalog_manager.cc:5495] T 6e599e3008f64ced9dcd0601e19e64b9 P 7f8375b6be764013974e4e579f7a6e80 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7f8375b6be764013974e4e579f7a6e80 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "7f8375b6be764013974e4e579f7a6e80" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf4adf2a6664d31af185aeb05846b9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7f8375b6be764013974e4e579f7a6e80" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34133 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "756f220b41d0464b947550c9f879f31b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44465 } health_report { overall_health: UNKNOWN } } }
W20260619 14:49:31.304395 15151 tablet.cc:2369] T 6e599e3008f64ced9dcd0601e19e64b9 P 0cf4adf2a6664d31af185aeb05846b9d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 14:49:31.309011 15019 tablet.cc:2369] T 6e599e3008f64ced9dcd0601e19e64b9 P 7f8375b6be764013974e4e579f7a6e80: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 14:49:31.310772 14881 tablet.cc:2369] T 6e599e3008f64ced9dcd0601e19e64b9 P 756f220b41d0464b947550c9f879f31b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:49:31.312301 15160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:31.307409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58560 (local address 127.0.98.2:34133)
0619 14:49:31.307563 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:31.307567 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:31.307581 (+ 14us) server_negotiation.cc:407] Connection header received
0619 14:49:31.307664 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:31.307667 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:31.307737 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:31.307851 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:49:31.308139 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:31.308820 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:31.309879 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:31.310105 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:31.310195 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:31.310723 (+ 528us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:49:31.311639 (+ 916us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:49:31.311853 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:31.311937 (+ 84us) server_negotiation.cc:299] Negotiation successful
0619 14:49:31.311999 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260619 14:49:31.319106 15104 raft_consensus.cc:1261] T 6e599e3008f64ced9dcd0601e19e64b9 P 0cf4adf2a6664d31af185aeb05846b9d [term 1 FOLLOWER]: Refusing update from remote peer 7f8375b6be764013974e4e579f7a6e80: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:49:31.319106 14834 raft_consensus.cc:1261] T 6e599e3008f64ced9dcd0601e19e64b9 P 756f220b41d0464b947550c9f879f31b [term 1 FOLLOWER]: Refusing update from remote peer 7f8375b6be764013974e4e579f7a6e80: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:49:31.320047 15173 consensus_queue.cc:1035] T 6e599e3008f64ced9dcd0601e19e64b9 P 7f8375b6be764013974e4e579f7a6e80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cf4adf2a6664d31af185aeb05846b9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:49:31.320343 15170 consensus_queue.cc:1035] T 6e599e3008f64ced9dcd0601e19e64b9 P 7f8375b6be764013974e4e579f7a6e80 [LEADER]: Connected to new peer: Peer: permanent_uuid: "756f220b41d0464b947550c9f879f31b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:49:31.330089 15175 mvcc.cc:205] Tried to move back new op lower bound from 7298582820113797120 to 7298582819897597952. Current Snapshot: MvccSnapshot[applied={T|T < 7298582820113797120}]
I20260619 14:49:31.330530 15176 mvcc.cc:205] Tried to move back new op lower bound from 7298582820113797120 to 7298582819897597952. Current Snapshot: MvccSnapshot[applied={T|T < 7298582820113797120}]
I20260619 14:49:31.337230 15177 mvcc.cc:205] Tried to move back new op lower bound from 7298582820113797120 to 7298582819897597952. Current Snapshot: MvccSnapshot[applied={T|T < 7298582820113797120 or (T in {7298582820113797120})}]
I20260619 14:49:31.340476 14687 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:37100:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:49:31.340703 14687 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:37100:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:49:31.343525 14687 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b6771a03d5c94a049845292e361e9034: Sending DeleteTablet for 3 replicas of tablet 6e599e3008f64ced9dcd0601e19e64b9
I20260619 14:49:31.344306 14813 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e599e3008f64ced9dcd0601e19e64b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:49:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38152
I20260619 14:49:31.344339 15084 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e599e3008f64ced9dcd0601e19e64b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:49:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44590
I20260619 14:49:31.344509 14952 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e599e3008f64ced9dcd0601e19e64b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:49:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58554
I20260619 14:49:31.345002 15185 tablet_replica.cc:330] T 6e599e3008f64ced9dcd0601e19e64b9 P 0cf4adf2a6664d31af185aeb05846b9d: stopping tablet replica
I20260619 14:49:31.345219 15186 tablet_replica.cc:330] T 6e599e3008f64ced9dcd0601e19e64b9 P 7f8375b6be764013974e4e579f7a6e80: stopping tablet replica
I20260619 14:49:31.345264 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 14730
I20260619 14:49:31.345654 15186 raft_consensus.cc:2229] T 6e599e3008f64ced9dcd0601e19e64b9 P 7f8375b6be764013974e4e579f7a6e80 [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:49:31.346290 15185 raft_consensus.cc:2229] T 6e599e3008f64ced9dcd0601e19e64b9 P 0cf4adf2a6664d31af185aeb05846b9d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:49:31.346789 15186 raft_consensus.cc:2258] T 6e599e3008f64ced9dcd0601e19e64b9 P 7f8375b6be764013974e4e579f7a6e80 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:49:31.347229 15185 raft_consensus.cc:2258] T 6e599e3008f64ced9dcd0601e19e64b9 P 0cf4adf2a6664d31af185aeb05846b9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:49:31.350212 15185 ts_tablet_manager.cc:1905] T 6e599e3008f64ced9dcd0601e19e64b9 P 0cf4adf2a6664d31af185aeb05846b9d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:49:31.351657 15186 ts_tablet_manager.cc:1905] T 6e599e3008f64ced9dcd0601e19e64b9 P 7f8375b6be764013974e4e579f7a6e80: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260619 14:49:31.353871 14679 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44465 (error 108)
I20260619 14:49:31.355314 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 14884
I20260619 14:49:31.355598 15185 ts_tablet_manager.cc:1918] T 6e599e3008f64ced9dcd0601e19e64b9 P 0cf4adf2a6664d31af185aeb05846b9d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:49:31.355716 15185 log.cc:1196] T 6e599e3008f64ced9dcd0601e19e64b9 P 0cf4adf2a6664d31af185aeb05846b9d: Deleting WAL directory at /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-2/wal/wals/6e599e3008f64ced9dcd0601e19e64b9
I20260619 14:49:31.356192 15185 ts_tablet_manager.cc:1939] T 6e599e3008f64ced9dcd0601e19e64b9 P 0cf4adf2a6664d31af185aeb05846b9d: Deleting consensus metadata
I20260619 14:49:31.357640 14676 catalog_manager.cc:4841] TS 0cf4adf2a6664d31af185aeb05846b9d (127.0.98.3:33209): tablet 6e599e3008f64ced9dcd0601e19e64b9 (table test-table [id=9a74b6a91b39418a895b2ce120ae4396]) successfully deleted
I20260619 14:49:31.357770 15159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:31.357347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44465 (local address 127.0.0.1:38158)
0619 14:49:31.357513 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:31.357597 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260619 14:49:31.357959 14679 catalog_manager.cc:4584] TS 756f220b41d0464b947550c9f879f31b (127.0.98.1:44465): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6e599e3008f64ced9dcd0601e19e64b9: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44465: connect: Connection refused (error 111)
I20260619 14:49:31.363181 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 15021
I20260619 14:49:31.363337 15159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:31.362930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34133 (local address 127.0.0.1:58566)
0619 14:49:31.363159 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:31.363230 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:34133: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
I20260619 14:49:31.370051 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 14659
I20260619 14:49:31.377369 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38293
--webserver_interface=127.0.98.62
--webserver_port=33531
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38293
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:49:31.501048 15188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:49:31.501602 15188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:49:31.501730 15188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:49:31.507753 15188 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:49:31.507860 15188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:49:31.507885 15188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:49:31.507905 15188 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:49:31.507925 15188 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:49:31.514997 15188 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38293
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38293
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=33531
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15188
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:49:31.516341 15188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:49:31.517568 15188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:49:31.528337 15194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:49:31.528237 15196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:49:31.528245 15193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:49:31.528748 15188 server_base.cc:1034] running on GCE node
I20260619 14:49:31.529497 15188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:49:31.530824 15188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:49:31.532025 15188 hybrid_clock.cc:648] HybridClock initialized: now 1781880571531998 us; error 63 us; skew 500 ppm
Jun 19 14:49:31 dist-test-slave-2sb1 krb5kdc[14111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880571, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:49:31.536441 15188 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 14:49:31.537906 15188 webserver.cc:460] Webserver started at http://127.0.98.62:33531/ using document root <none> and password file <none>
I20260619 14:49:31.538589 15188 fs_manager.cc:359] Metadata directory not provided
I20260619 14:49:31.538743 15188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:49:31.543622 15188 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:49:31.546661 15203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:31.548441 15188 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 14:49:31.548635 15188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/master-0/wal
uuid: "b6771a03d5c94a049845292e361e9034"
format_stamp: "Formatted at 2026-06-19 14:49:28 on dist-test-slave-2sb1"
server_key: "9a323fb970da1d2676ccf25ffee35f3d"
server_key_iv: "742b667a3ad03cebfdddd1ad034f72b9"
server_key_version: "encryptionkey@0"
I20260619 14:49:31.549368 15188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:49:31.580003 15188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:49:31.584368 15188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:49:31.584693 15188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:49:31.594384 15188 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38293
I20260619 14:49:31.594725 15255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38293 every 8 connection(s)
I20260619 14:49:31.595754 15188 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:49:31.595952 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 15188
I20260619 14:49:31.596551 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:44465
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:38293
--webserver_port=42965
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:49:31.601399 15256 sys_catalog.cc:263] Verifying existing consensus state
I20260619 14:49:31.603751 15256 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6771a03d5c94a049845292e361e9034: Bootstrap starting.
I20260619 14:49:31.624002 15256 log.cc:824] T 00000000000000000000000000000000 P b6771a03d5c94a049845292e361e9034: Log is configured to *not* fsync() on all Append() calls
I20260619 14:49:31.638481 15256 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6771a03d5c94a049845292e361e9034: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 14:49:31.639190 15256 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6771a03d5c94a049845292e361e9034: Bootstrap complete.
I20260619 14:49:31.644069 15256 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6771a03d5c94a049845292e361e9034 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6771a03d5c94a049845292e361e9034" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38293 } }
I20260619 14:49:31.645049 15256 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6771a03d5c94a049845292e361e9034 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6771a03d5c94a049845292e361e9034, State: Initialized, Role: FOLLOWER
I20260619 14:49:31.645855 15256 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6771a03d5c94a049845292e361e9034 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6771a03d5c94a049845292e361e9034" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38293 } }
I20260619 14:49:31.646050 15256 raft_consensus.cc:388] T 00000000000000000000000000000000 P b6771a03d5c94a049845292e361e9034 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:49:31.646153 15256 raft_consensus.cc:482] T 00000000000000000000000000000000 P b6771a03d5c94a049845292e361e9034 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:49:31.646294 15256 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6771a03d5c94a049845292e361e9034 [term 2 FOLLOWER]: Advancing to term 3
I20260619 14:49:31.648877 15256 raft_consensus.cc:504] T 00000000000000000000000000000000 P b6771a03d5c94a049845292e361e9034 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6771a03d5c94a049845292e361e9034" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38293 } }
I20260619 14:49:31.649281 15256 leader_election.cc:302] T 00000000000000000000000000000000 P b6771a03d5c94a049845292e361e9034 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b6771a03d5c94a049845292e361e9034; no voters:
I20260619 14:49:31.649688 15256 leader_election.cc:288] T 00000000000000000000000000000000 P b6771a03d5c94a049845292e361e9034 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 14:49:31.649798 15260 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b6771a03d5c94a049845292e361e9034 [term 3 FOLLOWER]: Leader election won for term 3
I20260619 14:49:31.650002 15260 raft_consensus.cc:686] T 00000000000000000000000000000000 P b6771a03d5c94a049845292e361e9034 [term 3 LEADER]: Becoming Leader. State: Replica: b6771a03d5c94a049845292e361e9034, State: Running, Role: LEADER
I20260619 14:49:31.650322 15260 consensus_queue.cc:237] T 00000000000000000000000000000000 P b6771a03d5c94a049845292e361e9034 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6771a03d5c94a049845292e361e9034" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38293 } }
I20260619 14:49:31.651134 15256 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6771a03d5c94a049845292e361e9034 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:49:31.651821 15261 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6771a03d5c94a049845292e361e9034 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "b6771a03d5c94a049845292e361e9034" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6771a03d5c94a049845292e361e9034" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38293 } } }
I20260619 14:49:31.651978 15261 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6771a03d5c94a049845292e361e9034 [sys.catalog]: This master's current role is: LEADER
I20260619 14:49:31.652594 15262 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6771a03d5c94a049845292e361e9034 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6771a03d5c94a049845292e361e9034. Latest consensus state: current_term: 3 leader_uuid: "b6771a03d5c94a049845292e361e9034" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6771a03d5c94a049845292e361e9034" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38293 } } }
I20260619 14:49:31.652837 15262 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6771a03d5c94a049845292e361e9034 [sys.catalog]: This master's current role is: LEADER
I20260619 14:49:31.653051 15268 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:49:31.659180 15268 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:49:31.660079 15268 catalog_manager.cc:1259] Loaded cluster ID: 3dd5a58639204044aeadf22270a16d68
I20260619 14:49:31.660161 15268 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260619 14:49:31.661398 15268 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f88b88b99ad google::LogMessage::Fail() at ??:0
@ 0x7f88b88bc88a google::LogMessage::SendToLog() at ??:0
@ 0x7f88b88b94de google::LogMessage::Flush() at ??:0
@ 0x7f88b88baa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f88c31f8fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f88c31f9d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f88c31f6579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f88c323a640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f88bab0b3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f88bab0bc83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f88bab0d3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f88baafc512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f88bc1406db start_thread at ??:0
@ 0x7f88b73b761f clone at ??:0
W20260619 14:49:31.756021 15258 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:49:31.756306 15258 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:49:31.756364 15258 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:49:31.761071 15258 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:49:31.761154 15258 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:49:31.761250 15258 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 14:49:31.767357 15258 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:44465
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=42965
--tserver_master_addrs=127.0.98.62:38293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15258
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:49:31.768534 15258 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:49:31.769620 15258 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:49:31.781392 15298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:49:31.781584 15258 server_base.cc:1034] running on GCE node
W20260619 14:49:31.781410 15300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:49:31.781482 15297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:49:31.782249 15258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:49:31.783104 15258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:49:31.784313 15258 hybrid_clock.cc:648] HybridClock initialized: now 1781880571784295 us; error 39 us; skew 500 ppm
Jun 19 14:49:31 dist-test-slave-2sb1 krb5kdc[14111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880571, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:49:31.788276 15258 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 14:49:31.789494 15258 webserver.cc:460] Webserver started at http://127.0.98.1:42965/ using document root <none> and password file <none>
I20260619 14:49:31.790205 15258 fs_manager.cc:359] Metadata directory not provided
I20260619 14:49:31.790323 15258 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:49:31.795481 15258 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 14:49:31.798262 15307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:31.799937 15258 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 14:49:31.800149 15258 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-0/wal
uuid: "756f220b41d0464b947550c9f879f31b"
format_stamp: "Formatted at 2026-06-19 14:49:29 on dist-test-slave-2sb1"
server_key: "d7ec8fd1d6cde41b65d369fc1d9e6ebc"
server_key_iv: "530e145e2ffdec8a5d393ce35f172ff2"
server_key_version: "encryptionkey@0"
I20260619 14:49:31.800868 15258 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:49:31.818008 15258 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:49:31.822156 15258 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:49:31.822402 15258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:49:31.823069 15258 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:49:31.824316 15314 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 14:49:31.829728 15258 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 14:49:31.829871 15258 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260619 14:49:31.829949 15258 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 14:49:31.832428 15258 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 14:49:31.832517 15258 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.000s sys 0.000s
I20260619 14:49:31.833168 15314 tablet_bootstrap.cc:492] T 6e599e3008f64ced9dcd0601e19e64b9 P 756f220b41d0464b947550c9f879f31b: Bootstrap starting.
I20260619 14:49:31.846844 15258 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44465
I20260619 14:49:31.846905 15421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44465 every 8 connection(s)
I20260619 14:49:31.848299 15258 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-0/data/info.pb
I20260619 14:49:31.853827 15314 log.cc:824] T 6e599e3008f64ced9dcd0601e19e64b9 P 756f220b41d0464b947550c9f879f31b: Log is configured to *not* fsync() on all Append() calls
I20260619 14:49:31.856740 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 15258
I20260619 14:49:31.857218 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:34133
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:38293
--webserver_port=33907
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:49:31.862114 15314 tablet_bootstrap.cc:492] T 6e599e3008f64ced9dcd0601e19e64b9 P 756f220b41d0464b947550c9f879f31b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 14:49:31.862542 15314 tablet_bootstrap.cc:492] T 6e599e3008f64ced9dcd0601e19e64b9 P 756f220b41d0464b947550c9f879f31b: Bootstrap complete.
I20260619 14:49:31.863708 15314 ts_tablet_manager.cc:1397] T 6e599e3008f64ced9dcd0601e19e64b9 P 756f220b41d0464b947550c9f879f31b: Time spent bootstrapping tablet: real 0.031s user 0.019s sys 0.010s
I20260619 14:49:31.866460 15314 raft_consensus.cc:348] T 6e599e3008f64ced9dcd0601e19e64b9 P 756f220b41d0464b947550c9f879f31b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf4adf2a6664d31af185aeb05846b9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33209 } } peers { permanent_uuid: "7f8375b6be764013974e4e579f7a6e80" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34133 } } peers { permanent_uuid: "756f220b41d0464b947550c9f879f31b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44465 } }
I20260619 14:49:31.866953 15314 raft_consensus.cc:729] T 6e599e3008f64ced9dcd0601e19e64b9 P 756f220b41d0464b947550c9f879f31b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 756f220b41d0464b947550c9f879f31b, State: Initialized, Role: FOLLOWER
I20260619 14:49:31.867486 15314 consensus_queue.cc:260] T 6e599e3008f64ced9dcd0601e19e64b9 P 756f220b41d0464b947550c9f879f31b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf4adf2a6664d31af185aeb05846b9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33209 } } peers { permanent_uuid: "7f8375b6be764013974e4e579f7a6e80" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34133 } } peers { permanent_uuid: "756f220b41d0464b947550c9f879f31b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44465 } }
I20260619 14:49:31.868515 15314 ts_tablet_manager.cc:1428] T 6e599e3008f64ced9dcd0601e19e64b9 P 756f220b41d0464b947550c9f879f31b: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
W20260619 14:49:31.971393 15424 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:49:31.850848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38293 (local address 127.0.98.1:55809)
0619 14:49:31.851544 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:31.851610 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0619 14:49:31.852708 (+ 1098us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:49:31.970389 (+117681us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38293: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":188,"threads_started":1}
W20260619 14:49:31.972052 15422 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38293 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:38293: Client connection negotiation failed: client connection to 127.0.98.62:38293: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 14:49:31.973160 15424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:31.972832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38293 (local address 127.0.98.1:36365)
0619 14:49:31.973022 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:31.973100 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260619 14:49:31.974045 15424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:31.973767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38293 (local address 127.0.98.1:36609)
0619 14:49:31.973913 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:31.973968 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260619 14:49:31.974231 15422 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 14:49:32.007817 15426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:49:32.008140 15426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:49:32.008235 15426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:49:32.013664 15426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:49:32.013795 15426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:49:32.013914 15426 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 14:49:32.020524 15426 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:34133
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=33907
--tserver_master_addrs=127.0.98.62:38293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15426
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:49:32.021865 15426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:49:32.023003 15426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:49:32.034665 15432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:49:32.034750 15426 server_base.cc:1034] running on GCE node
W20260619 14:49:32.034651 15435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:49:32.034653 15433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:49:32.035485 15426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:49:32.036324 15426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:49:32.037572 15426 hybrid_clock.cc:648] HybridClock initialized: now 1781880572037543 us; error 43 us; skew 500 ppm
Jun 19 14:49:32 dist-test-slave-2sb1 krb5kdc[14111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880572, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:49:32.040946 15426 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 14:49:32.041968 15426 webserver.cc:460] Webserver started at http://127.0.98.2:33907/ using document root <none> and password file <none>
I20260619 14:49:32.042644 15426 fs_manager.cc:359] Metadata directory not provided
I20260619 14:49:32.042716 15426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:49:32.047114 15426 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:49:32.049829 15442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:32.051644 15426 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 14:49:32.051816 15426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-1/wal
uuid: "7f8375b6be764013974e4e579f7a6e80"
format_stamp: "Formatted at 2026-06-19 14:49:29 on dist-test-slave-2sb1"
server_key: "75e11f0fa2159637d278bbc0f234423a"
server_key_iv: "dc2c124dcddeaee51cdca4d7b066e3dd"
server_key_version: "encryptionkey@0"
I20260619 14:49:32.052484 15426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:49:32.073223 15426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:49:32.077518 15426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:49:32.077762 15426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:49:32.078660 15426 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:49:32.080367 15449 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260619 14:49:32.084486 15449 ts_tablet_manager.cc:1852] T 6e599e3008f64ced9dcd0601e19e64b9 P 7f8375b6be764013974e4e579f7a6e80: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260619 14:49:32.085121 15449 ts_tablet_manager.cc:1905] T 6e599e3008f64ced9dcd0601e19e64b9 P 7f8375b6be764013974e4e579f7a6e80: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:49:32.089476 15449 ts_tablet_manager.cc:1918] T 6e599e3008f64ced9dcd0601e19e64b9 P 7f8375b6be764013974e4e579f7a6e80: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:49:32.089640 15449 log.cc:1196] T 6e599e3008f64ced9dcd0601e19e64b9 P 7f8375b6be764013974e4e579f7a6e80: Deleting WAL directory at /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-1/wal/wals/6e599e3008f64ced9dcd0601e19e64b9
I20260619 14:49:32.090060 15449 ts_tablet_manager.cc:1939] T 6e599e3008f64ced9dcd0601e19e64b9 P 7f8375b6be764013974e4e579f7a6e80: Deleting consensus metadata
I20260619 14:49:32.090427 15426 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260619 14:49:32.090524 15426 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s user 0.000s sys 0.000s
I20260619 14:49:32.090662 15426 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:49:32.090705 15426 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:32.102617 15426 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34133
I20260619 14:49:32.102643 15556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34133 every 8 connection(s)
I20260619 14:49:32.103605 15426 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-1/data/info.pb
I20260619 14:49:32.108094 15559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:32.106413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38293 (local address 127.0.98.2:51951)
0619 14:49:32.107040 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:32.107261 (+ 221us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":155,"threads_started":1}
W20260619 14:49:32.108865 15557 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38293 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:38293: Client connection negotiation failed: client connection to 127.0.98.62:38293: connect: Connection refused (error 111)
I20260619 14:49:32.109760 15559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:32.109379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38293 (local address 127.0.98.2:42183)
0619 14:49:32.109609 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:32.109677 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260619 14:49:32.109824 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 15426
I20260619 14:49:32.110322 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:33209
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:38293
--webserver_port=44395
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34313
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:49:32.110773 15559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:32.110497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38293 (local address 127.0.98.2:49495)
0619 14:49:32.110663 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:32.110710 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260619 14:49:32.111109 15557 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 14:49:32.237257 15560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:49:32.237566 15560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:49:32.237650 15560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:49:32.242050 15560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:49:32.242139 15560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:49:32.242262 15560 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 14:49:32.248054 15560 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34313
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:33209
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=44395
--tserver_master_addrs=127.0.98.62:38293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15560
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:49:32.249262 15560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:49:32.250249 15560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:49:32.261104 15568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:49:32.261121 15566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:49:32.261104 15565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:49:32.261165 15560 server_base.cc:1034] running on GCE node
I20260619 14:49:32.261780 15560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:49:32.262681 15560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:49:32.263960 15560 hybrid_clock.cc:648] HybridClock initialized: now 1781880572263947 us; error 34 us; skew 500 ppm
Jun 19 14:49:32 dist-test-slave-2sb1 krb5kdc[14111](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880572, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:49:32.267529 15560 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 14:49:32.268553 15560 webserver.cc:460] Webserver started at http://127.0.98.3:44395/ using document root <none> and password file <none>
I20260619 14:49:32.269189 15560 fs_manager.cc:359] Metadata directory not provided
I20260619 14:49:32.269258 15560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:49:32.274549 15560 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 14:49:32.277537 15575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:32.279300 15560 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:49:32.279448 15560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-2/wal
uuid: "0cf4adf2a6664d31af185aeb05846b9d"
format_stamp: "Formatted at 2026-06-19 14:49:29 on dist-test-slave-2sb1"
server_key: "67bcf70d36ed12b1ba5463a5836affec"
server_key_iv: "057d4659b4200470957a6190fdac9a5d"
server_key_version: "encryptionkey@0"
I20260619 14:49:32.280179 15560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:49:32.309950 15560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:49:32.313834 15560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:49:32.314081 15560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:49:32.314806 15560 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:49:32.315737 15560 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:49:32.315793 15560 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:32.315866 15560 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:49:32.315948 15560 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:32.326344 15560 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33209
I20260619 14:49:32.326362 15688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33209 every 8 connection(s)
I20260619 14:49:32.327461 15560 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781880383132139-392-0/minicluster-data/ts-2/data/info.pb
I20260619 14:49:32.328805 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 15560
I20260619 14:49:32.332063 15692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:32.329996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38293 (local address 127.0.98.3:49215)
0619 14:49:32.331028 (+ 1032us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:32.331145 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":814,"thread_start_us":603,"threads_started":1}
W20260619 14:49:32.332823 15689 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38293 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:38293: Client connection negotiation failed: client connection to 127.0.98.62:38293: connect: Connection refused (error 111)
I20260619 14:49:32.333808 15692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:32.333515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38293 (local address 127.0.98.3:51183)
0619 14:49:32.333681 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:32.333738 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260619 14:49:32.335258 15692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:32.334995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38293 (local address 127.0.98.3:51485)
0619 14:49:32.335140 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:32.335192 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260619 14:49:32.335577 15689 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 14:49:32.975567 15693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:32.975050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38293 (local address 127.0.98.1:48291)
0619 14:49:32.975371 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:32.975459 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":112,"threads_started":1}
I20260619 14:49:33.112409 15694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:33.111967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38293 (local address 127.0.98.2:33049)
0619 14:49:33.112253 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:33.112316 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":95,"threads_started":1}
I20260619 14:49:33.337033 15695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:33.336452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38293 (local address 127.0.98.3:47165)
0619 14:49:33.336810 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:33.336884 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":103,"threads_started":1}
I20260619 14:49:33.498036 15696 raft_consensus.cc:482] T 6e599e3008f64ced9dcd0601e19e64b9 P 756f220b41d0464b947550c9f879f31b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:49:33.498229 15696 raft_consensus.cc:504] T 6e599e3008f64ced9dcd0601e19e64b9 P 756f220b41d0464b947550c9f879f31b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf4adf2a6664d31af185aeb05846b9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33209 } } peers { permanent_uuid: "7f8375b6be764013974e4e579f7a6e80" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34133 } } peers { permanent_uuid: "756f220b41d0464b947550c9f879f31b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44465 } }
I20260619 14:49:33.499562 15696 leader_election.cc:288] T 6e599e3008f64ced9dcd0601e19e64b9 P 756f220b41d0464b947550c9f879f31b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0cf4adf2a6664d31af185aeb05846b9d (127.0.98.3:33209), 7f8375b6be764013974e4e579f7a6e80 (127.0.98.2:34133)
Jun 19 14:49:33 dist-test-slave-2sb1 krb5kdc[14111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880571, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Jun 19 14:49:33 dist-test-slave-2sb1 krb5kdc[14111](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880571, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260619 14:49:33.524932 15700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:33.500047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43827 (local address 127.0.98.2:34133)
0619 14:49:33.500452 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:33.500469 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:33.500531 (+ 62us) server_negotiation.cc:407] Connection header received
0619 14:49:33.501490 (+ 959us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:33.501497 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:33.501981 (+ 484us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:33.502342 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:49:33.504627 (+ 2285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:33.506650 (+ 2023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:33.508744 (+ 2094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:33.509562 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:33.509765 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:33.519295 (+ 9530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:49:33.519301 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:49:33.519317 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:49:33.519352 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:49:33.521750 (+ 2398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:33.522517 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:33.522527 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:33.522536 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:33.522695 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:33.523151 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:33.523156 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:33.523157 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:33.523621 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:49:33.523780 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:33.524364 (+ 584us) server_negotiation.cc:299] Negotiation successful
0619 14:49:33.524703 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":106,"threads_started":1}
I20260619 14:49:33.524927 15697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:33.499751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33209 (local address 127.0.98.1:39583)
0619 14:49:33.500083 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:33.500102 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 14:49:33.500235 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:49:33.502551 (+ 2316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:49:33.502559 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:49:33.503295 (+ 736us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:49:33.504325 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:33.504354 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:33.506836 (+ 2482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:33.506839 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:33.508556 (+ 1717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:33.508571 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:33.509726 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:33.509731 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:33.509996 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:33.516271 (+ 6275us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:49:33.516308 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:49:33.519067 (+ 2759us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:49:33.521956 (+ 2889us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:49:33.521960 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:49:33.521962 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:49:33.522447 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:49:33.522892 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:49:33.522894 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:49:33.522896 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:49:33.523049 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:49:33.523761 (+ 712us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:49:33.523769 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:49:33.524035 (+ 266us) client_negotiation.cc:769] Sending connection context
0619 14:49:33.524236 (+ 201us) client_negotiation.cc:240] Negotiation successful
0619 14:49:33.524723 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"mutex_wait_us":460,"spinlock_wait_cycles":51968,"thread_start_us":91,"threads_started":1}
I20260619 14:49:33.525034 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:33.499978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39583 (local address 127.0.98.3:33209)
0619 14:49:33.500489 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:33.500509 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:33.500539 (+ 30us) server_negotiation.cc:407] Connection header received
0619 14:49:33.501481 (+ 942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:33.501490 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:33.501981 (+ 491us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:33.502342 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:49:33.504627 (+ 2285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:33.506650 (+ 2023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:33.508744 (+ 2094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:33.509562 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:33.509765 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:33.519361 (+ 9596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:49:33.519367 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:49:33.519379 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:49:33.519415 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:49:33.521771 (+ 2356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:33.522607 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:33.522612 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:33.522616 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:33.522744 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:33.523198 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:33.523201 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:33.523203 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:33.523622 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:49:33.523764 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:33.524371 (+ 607us) server_negotiation.cc:299] Negotiation successful
0619 14:49:33.524818 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":111,"threads_started":1}
I20260619 14:49:33.525035 15698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:33.499897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34133 (local address 127.0.98.1:43827)
0619 14:49:33.500242 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:33.500256 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:49:33.500359 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:49:33.502551 (+ 2192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:49:33.502559 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:49:33.503295 (+ 736us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:49:33.504325 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:33.504354 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:33.506830 (+ 2476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:33.506835 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:33.508556 (+ 1721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:33.508571 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:33.509726 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:33.509731 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:33.510093 (+ 362us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:33.516271 (+ 6178us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:49:33.516308 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:49:33.518967 (+ 2659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:49:33.521932 (+ 2965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:49:33.521941 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:49:33.521957 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:49:33.522369 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:49:33.522832 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:49:33.522835 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:49:33.522838 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:49:33.523043 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:49:33.523786 (+ 743us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:49:33.523789 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:49:33.524036 (+ 247us) client_negotiation.cc:769] Sending connection context
0619 14:49:33.524238 (+ 202us) client_negotiation.cc:240] Negotiation successful
0619 14:49:33.524812 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"mutex_wait_us":709,"spinlock_wait_cycles":107904,"thread_start_us":73,"threads_started":1}
W20260619 14:49:33.528118 15309 leader_election.cc:341] T 6e599e3008f64ced9dcd0601e19e64b9 P 756f220b41d0464b947550c9f879f31b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7f8375b6be764013974e4e579f7a6e80 (127.0.98.2:34133): Not found: Tablet not found: 6e599e3008f64ced9dcd0601e19e64b9
W20260619 14:49:33.528378 15308 leader_election.cc:341] T 6e599e3008f64ced9dcd0601e19e64b9 P 756f220b41d0464b947550c9f879f31b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0cf4adf2a6664d31af185aeb05846b9d (127.0.98.3:33209): Not found: Tablet not found: 6e599e3008f64ced9dcd0601e19e64b9
I20260619 14:49:33.528460 15308 leader_election.cc:302] T 6e599e3008f64ced9dcd0601e19e64b9 P 756f220b41d0464b947550c9f879f31b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 756f220b41d0464b947550c9f879f31b; no voters: 0cf4adf2a6664d31af185aeb05846b9d, 7f8375b6be764013974e4e579f7a6e80
I20260619 14:49:33.528743 15696 raft_consensus.cc:2726] T 6e599e3008f64ced9dcd0601e19e64b9 P 756f220b41d0464b947550c9f879f31b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 14:49:33.977234 15697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:33.976806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38293 (local address 127.0.98.1:37211)
0619 14:49:33.977080 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:33.977143 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175}
I20260619 14:49:34.113950 15701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:34.113325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38293 (local address 127.0.98.2:50929)
0619 14:49:34.113622 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:34.113715 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":88,"threads_started":1}
I20260619 14:49:34.338451 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:34.338023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38293 (local address 127.0.98.3:39423)
0619 14:49:34.338270 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:34.338335 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":105,"threads_started":1}
I20260619 14:49:34.978693 15703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:34.978202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38293 (local address 127.0.98.1:54739)
0619 14:49:34.978528 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:34.978595 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":86,"threads_started":1}
I20260619 14:49:35.086728 15704 raft_consensus.cc:482] T 6e599e3008f64ced9dcd0601e19e64b9 P 756f220b41d0464b947550c9f879f31b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:49:35.086997 15704 raft_consensus.cc:504] T 6e599e3008f64ced9dcd0601e19e64b9 P 756f220b41d0464b947550c9f879f31b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf4adf2a6664d31af185aeb05846b9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33209 } } peers { permanent_uuid: "7f8375b6be764013974e4e579f7a6e80" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34133 } } peers { permanent_uuid: "756f220b41d0464b947550c9f879f31b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44465 } }
I20260619 14:49:35.087574 15704 leader_election.cc:288] T 6e599e3008f64ced9dcd0601e19e64b9 P 756f220b41d0464b947550c9f879f31b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0cf4adf2a6664d31af185aeb05846b9d (127.0.98.3:33209), 7f8375b6be764013974e4e579f7a6e80 (127.0.98.2:34133)
W20260619 14:49:35.088632 15308 leader_election.cc:341] T 6e599e3008f64ced9dcd0601e19e64b9 P 756f220b41d0464b947550c9f879f31b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0cf4adf2a6664d31af185aeb05846b9d (127.0.98.3:33209): Not found: Tablet not found: 6e599e3008f64ced9dcd0601e19e64b9
W20260619 14:49:35.088802 15309 leader_election.cc:341] T 6e599e3008f64ced9dcd0601e19e64b9 P 756f220b41d0464b947550c9f879f31b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7f8375b6be764013974e4e579f7a6e80 (127.0.98.2:34133): Not found: Tablet not found: 6e599e3008f64ced9dcd0601e19e64b9
I20260619 14:49:35.089030 15309 leader_election.cc:302] T 6e599e3008f64ced9dcd0601e19e64b9 P 756f220b41d0464b947550c9f879f31b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 756f220b41d0464b947550c9f879f31b; no voters: 0cf4adf2a6664d31af185aeb05846b9d, 7f8375b6be764013974e4e579f7a6e80
I20260619 14:49:35.089218 15704 raft_consensus.cc:2726] T 6e599e3008f64ced9dcd0601e19e64b9 P 756f220b41d0464b947550c9f879f31b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 14:49:35.115298 15705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:35.114874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38293 (local address 127.0.98.2:52121)
0619 14:49:35.115130 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:35.115186 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":75,"threads_started":1}
I20260619 14:49:35.340116 15706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:35.339454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38293 (local address 127.0.98.3:40581)
0619 14:49:35.339822 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:35.339997 (+ 175us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":116,"threads_started":1}
I20260619 14:49:35.980453 15707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:35.979787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38293 (local address 127.0.98.1:59823)
0619 14:49:35.980219 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:35.980285 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":88,"threads_started":1}
I20260619 14:49:36.117020 15708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:36.116393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38293 (local address 127.0.98.2:42009)
0619 14:49:36.116769 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:36.116861 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":107,"threads_started":1}
I20260619 14:49:36.341567 15709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:36.341064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38293 (local address 127.0.98.3:37215)
0619 14:49:36.341375 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:36.341452 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":86,"threads_started":1}
I20260619 14:49:36.982424 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:36.981721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38293 (local address 127.0.98.1:59315)
0619 14:49:36.982152 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:36.982248 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":105,"threads_started":1}
I20260619 14:49:37.118579 15711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:37.118006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38293 (local address 127.0.98.2:42467)
0619 14:49:37.118422 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:37.118481 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":122,"threads_started":1}
I20260619 14:49:37.343140 15712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:37.342578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38293 (local address 127.0.98.3:43691)
0619 14:49:37.342926 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:37.343008 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":93,"threads_started":1}
I20260619 14:49:37.984045 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:37.983408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38293 (local address 127.0.98.1:56497)
0619 14:49:37.983830 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:37.983914 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":176,"threads_started":1}
I20260619 14:49:38.120211 15714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:38.119731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38293 (local address 127.0.98.2:56773)
0619 14:49:38.120046 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:38.120112 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":103,"threads_started":1}
I20260619 14:49:38.344766 15715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:38.344210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38293 (local address 127.0.98.3:53155)
0619 14:49:38.344527 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:38.344593 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":100,"threads_started":1}
I20260619 14:49:38.346597 15716 raft_consensus.cc:482] T 6e599e3008f64ced9dcd0601e19e64b9 P 756f220b41d0464b947550c9f879f31b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:49:38.346745 15716 raft_consensus.cc:504] T 6e599e3008f64ced9dcd0601e19e64b9 P 756f220b41d0464b947550c9f879f31b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf4adf2a6664d31af185aeb05846b9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33209 } } peers { permanent_uuid: "7f8375b6be764013974e4e579f7a6e80" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34133 } } peers { permanent_uuid: "756f220b41d0464b947550c9f879f31b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44465 } }
I20260619 14:49:38.347213 15716 leader_election.cc:288] T 6e599e3008f64ced9dcd0601e19e64b9 P 756f220b41d0464b947550c9f879f31b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0cf4adf2a6664d31af185aeb05846b9d (127.0.98.3:33209), 7f8375b6be764013974e4e579f7a6e80 (127.0.98.2:34133)
W20260619 14:49:38.348169 15309 leader_election.cc:341] T 6e599e3008f64ced9dcd0601e19e64b9 P 756f220b41d0464b947550c9f879f31b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7f8375b6be764013974e4e579f7a6e80 (127.0.98.2:34133): Not found: Tablet not found: 6e599e3008f64ced9dcd0601e19e64b9
W20260619 14:49:38.348277 15308 leader_election.cc:341] T 6e599e3008f64ced9dcd0601e19e64b9 P 756f220b41d0464b947550c9f879f31b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0cf4adf2a6664d31af185aeb05846b9d (127.0.98.3:33209): Not found: Tablet not found: 6e599e3008f64ced9dcd0601e19e64b9
I20260619 14:49:38.348335 15308 leader_election.cc:302] T 6e599e3008f64ced9dcd0601e19e64b9 P 756f220b41d0464b947550c9f879f31b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 756f220b41d0464b947550c9f879f31b; no voters: 0cf4adf2a6664d31af185aeb05846b9d, 7f8375b6be764013974e4e579f7a6e80
I20260619 14:49:38.348547 15716 raft_consensus.cc:2726] T 6e599e3008f64ced9dcd0601e19e64b9 P 756f220b41d0464b947550c9f879f31b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 14:49:38.985630 15717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:38.985075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38293 (local address 127.0.98.1:41105)
0619 14:49:38.985365 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:38.985436 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":130,"threads_started":1}
I20260619 14:49:39.121873 15718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:39.121307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38293 (local address 127.0.98.2:37821)
0619 14:49:39.121710 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:39.121777 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":141,"threads_started":1}
I20260619 14:49:39.346205 15719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:39.345671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38293 (local address 127.0.98.3:35685)
0619 14:49:39.346035 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:39.346106 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":77,"threads_started":1}
I20260619 14:49:39.987133 15720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:39.986551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38293 (local address 127.0.98.1:60219)
0619 14:49:39.986852 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:39.987031 (+ 179us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":79,"threads_started":1}
I20260619 14:49:40.123549 15721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:40.123049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38293 (local address 127.0.98.2:48789)
0619 14:49:40.123360 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:40.123434 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":82,"threads_started":1}
I20260619 14:49:40.347725 15722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:40.347208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38293 (local address 127.0.98.3:38807)
0619 14:49:40.347560 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:40.347628 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":98,"threads_started":1}
I20260619 14:49:40.988587 15723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:40.988108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38293 (local address 127.0.98.1:53895)
0619 14:49:40.988410 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:40.988475 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":109,"threads_started":1}
I20260619 14:49:41.125428 15724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:41.124828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38293 (local address 127.0.98.2:35665)
0619 14:49:41.125276 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:41.125327 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":108,"threads_started":1}
I20260619 14:49:41.349721 15725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:41.349021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38293 (local address 127.0.98.3:33201)
0619 14:49:41.349518 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:41.349594 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":117,"threads_started":1}
I20260619 14:49:41.990079 15726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:41.989628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38293 (local address 127.0.98.1:52947)
0619 14:49:41.989901 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:41.989979 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":80,"threads_started":1}
I20260619 14:49:42.126811 15727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:42.126323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38293 (local address 127.0.98.2:50283)
0619 14:49:42.126649 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:42.126715 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":102,"threads_started":1}
I20260619 14:49:42.294050 15728 raft_consensus.cc:482] T 6e599e3008f64ced9dcd0601e19e64b9 P 756f220b41d0464b947550c9f879f31b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:49:42.294191 15728 raft_consensus.cc:504] T 6e599e3008f64ced9dcd0601e19e64b9 P 756f220b41d0464b947550c9f879f31b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cf4adf2a6664d31af185aeb05846b9d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33209 } } peers { permanent_uuid: "7f8375b6be764013974e4e579f7a6e80" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34133 } } peers { permanent_uuid: "756f220b41d0464b947550c9f879f31b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44465 } }
I20260619 14:49:42.294595 15728 leader_election.cc:288] T 6e599e3008f64ced9dcd0601e19e64b9 P 756f220b41d0464b947550c9f879f31b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0cf4adf2a6664d31af185aeb05846b9d (127.0.98.3:33209), 7f8375b6be764013974e4e579f7a6e80 (127.0.98.2:34133)
W20260619 14:49:42.295601 15308 leader_election.cc:341] T 6e599e3008f64ced9dcd0601e19e64b9 P 756f220b41d0464b947550c9f879f31b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0cf4adf2a6664d31af185aeb05846b9d (127.0.98.3:33209): Not found: Tablet not found: 6e599e3008f64ced9dcd0601e19e64b9
W20260619 14:49:42.295727 15309 leader_election.cc:341] T 6e599e3008f64ced9dcd0601e19e64b9 P 756f220b41d0464b947550c9f879f31b [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7f8375b6be764013974e4e579f7a6e80 (127.0.98.2:34133): Not found: Tablet not found: 6e599e3008f64ced9dcd0601e19e64b9
I20260619 14:49:42.295881 15309 leader_election.cc:302] T 6e599e3008f64ced9dcd0601e19e64b9 P 756f220b41d0464b947550c9f879f31b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 756f220b41d0464b947550c9f879f31b; no voters: 0cf4adf2a6664d31af185aeb05846b9d, 7f8375b6be764013974e4e579f7a6e80
I20260619 14:49:42.296165 15728 raft_consensus.cc:2726] T 6e599e3008f64ced9dcd0601e19e64b9 P 756f220b41d0464b947550c9f879f31b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 14:49:42.351205 15729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:42.350689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38293 (local address 127.0.98.3:35523)
0619 14:49:42.351018 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:42.351084 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":97,"threads_started":1}
I20260619 14:49:42.991535 15730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:42.991037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38293 (local address 127.0.98.1:48537)
0619 14:49:42.991340 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:42.991424 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":78,"threads_started":1}
I20260619 14:49:43.128461 15731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:43.127922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38293 (local address 127.0.98.2:53035)
0619 14:49:43.128291 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:43.128354 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":76,"threads_started":1}
I20260619 14:49:43.352939 15732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:43.352310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38293 (local address 127.0.98.3:41093)
0619 14:49:43.352766 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:43.352835 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":170,"threads_started":1}
I20260619 14:49:43.993148 15733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:43.992576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38293 (local address 127.0.98.1:36921)
0619 14:49:43.992956 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:43.993028 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":140,"threads_started":1}
I20260619 14:49:44.130019 15734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:44.129479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38293 (local address 127.0.98.2:47281)
0619 14:49:44.129840 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:44.129911 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":97,"threads_started":1}
I20260619 14:49:44.354578 15735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:44.354016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38293 (local address 127.0.98.3:45295)
0619 14:49:44.354384 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:44.354478 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":109,"threads_started":1}
I20260619 14:49:44.994973 15736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:44.994260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38293 (local address 127.0.98.1:55927)
0619 14:49:44.994713 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:44.994827 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":125,"threads_started":1}
I20260619 14:49:45.131743 15737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:45.131207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38293 (local address 127.0.98.2:49975)
0619 14:49:45.131565 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:45.131642 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":80,"threads_started":1}
I20260619 14:49:45.356413 15738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:45.355882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38293 (local address 127.0.98.3:35729)
0619 14:49:45.356253 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:45.356319 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":99,"threads_started":1}
I20260619 14:49:45.996578 15739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:45.996004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38293 (local address 127.0.98.1:45085)
0619 14:49:45.996372 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:45.996444 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":90,"threads_started":1}
I20260619 14:49:46.133306 15740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:46.132843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38293 (local address 127.0.98.2:44993)
0619 14:49:46.133120 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:46.133194 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":82,"threads_started":1}
I20260619 14:49:46.358106 15741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:46.357609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38293 (local address 127.0.98.3:40453)
0619 14:49:46.357931 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:46.357996 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":79,"threads_started":1}
I20260619 14:49:46.998567 15742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:46.998004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38293 (local address 127.0.98.1:37751)
0619 14:49:46.998400 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:46.998465 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":106,"threads_started":1}
I20260619 14:49:47.134984 15743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:47.134389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38293 (local address 127.0.98.2:54249)
0619 14:49:47.134771 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:47.134852 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:38293: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":78,"threads_started":1}
I20260619 14:49:47.337112 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 15258
I20260619 14:49:47.343186 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 15426
I20260619 14:49:47.349210 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 15560
2026-06-19T14:49:47Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19795 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:49:47 dist-test-slave-2sb1 krb5kdc[15746](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:49:47 dist-test-slave-2sb1 krb5kdc[15746](info): set up 2 sockets
Jun 19 14:49:47 dist-test-slave-2sb1 krb5kdc[15746](info): commencing operation
krb5kdc: starting...
W20260619 14:49:48.449277 392 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/krb5kdc/test-user.keytab.
Jun 19 14:49:48 dist-test-slave-2sb1 krb5kdc[15746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880588, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:49:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:49:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 14:49:48.597924 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39877
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39877
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:49:48.721571 15759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:49:48.721879 15759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:49:48.721921 15759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:49:48.726444 15759 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 14:49:48.726527 15759 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 14:49:48.726544 15759 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 14:49:48.726558 15759 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:49:48.726578 15759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:49:48.726598 15759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:49:48.726614 15759 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:49:48.726676 15759 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:49:48.732388 15759 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39877
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15759
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:49:48.733615 15759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:49:48.734766 15759 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:49:48.744807 15764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:49:48.744935 15767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:49:48.744916 15759 server_base.cc:1034] running on GCE node
W20260619 14:49:48.744915 15765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:49:48.746086 15759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:49:48.747486 15759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:49:48.748778 15759 hybrid_clock.cc:648] HybridClock initialized: now 1781880588748726 us; error 64 us; skew 500 ppm
Jun 19 14:49:48 dist-test-slave-2sb1 krb5kdc[15746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880588, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:49:48.752238 15759 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 14:49:48.753367 15759 webserver.cc:460] Webserver started at http://127.0.98.62:35415/ using document root <none> and password file <none>
I20260619 14:49:48.753969 15759 fs_manager.cc:359] Metadata directory not provided
I20260619 14:49:48.754076 15759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:49:48.754330 15759 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:49:48.756577 15759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/master-0/data/instance:
uuid: "85f59d8a5c854579995d8ebfc3501da5"
format_stamp: "Formatted at 2026-06-19 14:49:48 on dist-test-slave-2sb1"
server_key: "bdf7dd6b4385ff46418d8542894d08c3"
server_key_iv: "0d22b10314e11f96a24cb8d704875a66"
server_key_version: "encryptionkey@0"
I20260619 14:49:48.757313 15759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/master-0/wal/instance:
uuid: "85f59d8a5c854579995d8ebfc3501da5"
format_stamp: "Formatted at 2026-06-19 14:49:48 on dist-test-slave-2sb1"
server_key: "bdf7dd6b4385ff46418d8542894d08c3"
server_key_iv: "0d22b10314e11f96a24cb8d704875a66"
server_key_version: "encryptionkey@0"
I20260619 14:49:48.761642 15759 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 14:49:48.764799 15774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:48.766266 15759 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:49:48.766446 15759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/master-0/wal
uuid: "85f59d8a5c854579995d8ebfc3501da5"
format_stamp: "Formatted at 2026-06-19 14:49:48 on dist-test-slave-2sb1"
server_key: "bdf7dd6b4385ff46418d8542894d08c3"
server_key_iv: "0d22b10314e11f96a24cb8d704875a66"
server_key_version: "encryptionkey@0"
I20260619 14:49:48.766644 15759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:49:48.787542 15759 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:49:48.792248 15759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:49:48.792511 15759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:49:48.801558 15759 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39877
I20260619 14:49:48.801980 15826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39877 every 8 connection(s)
I20260619 14:49:48.802680 15759 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:49:48.805995 15827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:49:48.806053 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 15759
I20260619 14:49:48.806196 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/master-0/wal/instance
I20260619 14:49:48.806484 392 external_mini_cluster.cc:1428] Setting key 97ddf74169afd56c6ba7af68a36722e9
I20260619 14:49:48.814003 15827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85f59d8a5c854579995d8ebfc3501da5: Bootstrap starting.
I20260619 14:49:48.816891 15827 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 85f59d8a5c854579995d8ebfc3501da5: Neither blocks nor log segments found. Creating new log.
Jun 19 14:49:48 dist-test-slave-2sb1 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880588, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:49:48.817977 15827 log.cc:824] T 00000000000000000000000000000000 P 85f59d8a5c854579995d8ebfc3501da5: Log is configured to *not* fsync() on all Append() calls
I20260619 14:49:48.820690 15827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85f59d8a5c854579995d8ebfc3501da5: No bootstrap required, opened a new log
I20260619 14:49:48.824241 15830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:48.808094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57726 (local address 127.0.98.62:39877)
0619 14:49:48.808604 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:48.808623 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:48.808697 (+ 74us) server_negotiation.cc:407] Connection header received
0619 14:49:48.809869 (+ 1172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:48.809897 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:48.810378 (+ 481us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:48.810775 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:49:48.811610 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:48.813199 (+ 1589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:48.814312 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:48.815012 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:48.815187 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:48.818622 (+ 3435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:49:48.818630 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:49:48.818651 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:49:48.818680 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:49:48.820775 (+ 2095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:48.821377 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:48.821383 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:48.821390 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:48.821541 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:48.821817 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:48.821821 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:48.821823 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:48.822358 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:49:48.822560 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:48.822786 (+ 226us) server_negotiation.cc:299] Negotiation successful
0619 14:49:48.823164 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":172,"threads_started":1}
I20260619 14:49:48.825435 15827 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85f59d8a5c854579995d8ebfc3501da5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85f59d8a5c854579995d8ebfc3501da5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39877 } }
I20260619 14:49:48.825690 15827 raft_consensus.cc:374] T 00000000000000000000000000000000 P 85f59d8a5c854579995d8ebfc3501da5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:49:48.825767 15827 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85f59d8a5c854579995d8ebfc3501da5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f59d8a5c854579995d8ebfc3501da5, State: Initialized, Role: FOLLOWER
I20260619 14:49:48.826207 15827 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85f59d8a5c854579995d8ebfc3501da5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85f59d8a5c854579995d8ebfc3501da5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39877 } }
I20260619 14:49:48.826337 15827 raft_consensus.cc:388] T 00000000000000000000000000000000 P 85f59d8a5c854579995d8ebfc3501da5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:49:48.826426 15827 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85f59d8a5c854579995d8ebfc3501da5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:49:48.826522 15827 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85f59d8a5c854579995d8ebfc3501da5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:49:48.827607 15827 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85f59d8a5c854579995d8ebfc3501da5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85f59d8a5c854579995d8ebfc3501da5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39877 } }
I20260619 14:49:48.827976 15827 leader_election.cc:302] T 00000000000000000000000000000000 P 85f59d8a5c854579995d8ebfc3501da5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85f59d8a5c854579995d8ebfc3501da5; no voters:
I20260619 14:49:48.828406 15827 leader_election.cc:288] T 00000000000000000000000000000000 P 85f59d8a5c854579995d8ebfc3501da5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:49:48.828713 15832 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 85f59d8a5c854579995d8ebfc3501da5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:49:48.828974 15832 raft_consensus.cc:686] T 00000000000000000000000000000000 P 85f59d8a5c854579995d8ebfc3501da5 [term 1 LEADER]: Becoming Leader. State: Replica: 85f59d8a5c854579995d8ebfc3501da5, State: Running, Role: LEADER
I20260619 14:49:48.829438 15832 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85f59d8a5c854579995d8ebfc3501da5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85f59d8a5c854579995d8ebfc3501da5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39877 } }
I20260619 14:49:48.829804 15827 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85f59d8a5c854579995d8ebfc3501da5 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:49:48.831519 15834 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85f59d8a5c854579995d8ebfc3501da5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85f59d8a5c854579995d8ebfc3501da5. Latest consensus state: current_term: 1 leader_uuid: "85f59d8a5c854579995d8ebfc3501da5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85f59d8a5c854579995d8ebfc3501da5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39877 } } }
I20260619 14:49:48.831579 15833 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85f59d8a5c854579995d8ebfc3501da5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "85f59d8a5c854579995d8ebfc3501da5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85f59d8a5c854579995d8ebfc3501da5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39877 } } }
I20260619 14:49:48.831668 15834 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85f59d8a5c854579995d8ebfc3501da5 [sys.catalog]: This master's current role is: LEADER
I20260619 14:49:48.831708 15833 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85f59d8a5c854579995d8ebfc3501da5 [sys.catalog]: This master's current role is: LEADER
I20260619 14:49:48.832154 15844 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:49:48.835626 15844 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:49:48.842334 15844 catalog_manager.cc:1347] Generated new cluster ID: 3837edbdb3314120983e94b9eea509df
I20260619 14:49:48.842433 15844 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:49:48.860719 15844 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:49:48.862088 15844 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:49:48.871627 15844 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 85f59d8a5c854579995d8ebfc3501da5: Generated new TSK 0
I20260619 14:49:48.872633 15844 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 14:49:48.909725 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39877
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:49:49.034432 15853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:49:49.034747 15853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:49:49.034794 15853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:49:49.039215 15853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:49:49.039300 15853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:49:49.039367 15853 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 14:49:49.045351 15853 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39877
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15853
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:49:49.046602 15853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:49:49.047727 15853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:49:49.063304 15859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:49:49.063370 15861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:49:49.063767 15858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:49:49.063928 15853 server_base.cc:1034] running on GCE node
I20260619 14:49:49.064458 15853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:49:49.065630 15853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:49:49.066926 15853 hybrid_clock.cc:648] HybridClock initialized: now 1781880589066877 us; error 71 us; skew 500 ppm
Jun 19 14:49:49 dist-test-slave-2sb1 krb5kdc[15746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880589, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:49:49.071345 15853 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 14:49:49.072856 15853 webserver.cc:460] Webserver started at http://127.0.98.1:32955/ using document root <none> and password file <none>
I20260619 14:49:49.073704 15853 fs_manager.cc:359] Metadata directory not provided
I20260619 14:49:49.073814 15853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:49:49.074141 15853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:49:49.078086 15853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-0/data/instance:
uuid: "4bf51bda973949fcb3568cb66538fcdd"
format_stamp: "Formatted at 2026-06-19 14:49:49 on dist-test-slave-2sb1"
server_key: "abb9f876c53f30d1f950e3f76a32d324"
server_key_iv: "f4a116fee231edd0276c783399b9b061"
server_key_version: "encryptionkey@0"
I20260619 14:49:49.079059 15853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-0/wal/instance:
uuid: "4bf51bda973949fcb3568cb66538fcdd"
format_stamp: "Formatted at 2026-06-19 14:49:49 on dist-test-slave-2sb1"
server_key: "abb9f876c53f30d1f950e3f76a32d324"
server_key_iv: "f4a116fee231edd0276c783399b9b061"
server_key_version: "encryptionkey@0"
I20260619 14:49:49.085000 15853 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260619 14:49:49.088413 15868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:49.090128 15853 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:49:49.090282 15853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-0/wal
uuid: "4bf51bda973949fcb3568cb66538fcdd"
format_stamp: "Formatted at 2026-06-19 14:49:49 on dist-test-slave-2sb1"
server_key: "abb9f876c53f30d1f950e3f76a32d324"
server_key_iv: "f4a116fee231edd0276c783399b9b061"
server_key_version: "encryptionkey@0"
I20260619 14:49:49.090401 15853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:49:49.118897 15853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:49:49.122848 15853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:49:49.123209 15853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:49:49.123939 15853 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:49:49.125322 15853 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:49:49.125391 15853 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:49.125440 15853 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:49:49.125459 15853 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:49.135600 15853 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43063
I20260619 14:49:49.135692 15981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43063 every 8 connection(s)
I20260619 14:49:49.136694 15853 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-0/data/info.pb
I20260619 14:49:49.139217 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 15853
I20260619 14:49:49.139370 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-0/wal/instance
I20260619 14:49:49.139662 392 external_mini_cluster.cc:1428] Setting key 8193d25cef151afbd37ac9dd4018f90e
Jun 19 14:49:49 dist-test-slave-2sb1 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880589, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 14:49:49.155866 15830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:49.139059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45397 (local address 127.0.98.62:39877)
0619 14:49:49.139223 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:49.139228 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:49.140258 (+ 1030us) server_negotiation.cc:407] Connection header received
0619 14:49:49.141895 (+ 1637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:49.141898 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:49.141943 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:49.142035 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:49:49.143765 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:49.144922 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:49.147420 (+ 2498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:49.147698 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:49.147809 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:49.150805 (+ 2996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:49:49.150809 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:49:49.150813 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:49:49.150850 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:49:49.152958 (+ 2108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:49.153833 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:49.153837 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:49.153838 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:49.153998 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:49.154461 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:49.154465 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:49.154466 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:49.154757 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:49:49.154855 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:49.155576 (+ 721us) server_negotiation.cc:299] Negotiation successful
0619 14:49:49.155717 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 14:49:49.156975 15984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:49.139544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39877 (local address 127.0.98.1:45397)
0619 14:49:49.140092 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:49.140151 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 14:49:49.141623 (+ 1472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:49:49.142270 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:49:49.142282 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:49:49.142843 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:49:49.143584 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:49.143601 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:49.145166 (+ 1565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:49.145173 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:49.147234 (+ 2061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:49.147256 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:49.147912 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:49.147917 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:49.148134 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:49.148898 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:49:49.148916 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:49:49.150517 (+ 1601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:49:49.153113 (+ 2596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:49:49.153120 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:49:49.153138 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:49:49.153564 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:49:49.154141 (+ 577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:49:49.154143 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:49:49.154145 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:49:49.154331 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:49:49.154903 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:49:49.154910 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:49:49.155284 (+ 374us) client_negotiation.cc:769] Sending connection context
0619 14:49:49.155470 (+ 186us) client_negotiation.cc:240] Negotiation successful
0619 14:49:49.155791 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":167,"threads_started":1}
I20260619 14:49:49.159015 15982 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39877
I20260619 14:49:49.159368 15982 heartbeater.cc:464] Registering TS with master...
I20260619 14:49:49.160087 15982 heartbeater.cc:511] Master 127.0.98.62:39877 requested a full tablet report, sending...
I20260619 14:49:49.162169 15791 ts_manager.cc:194] Registered new tserver with Master: 4bf51bda973949fcb3568cb66538fcdd (127.0.98.1:43063)
I20260619 14:49:49.163434 15791 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45397
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 14:49:49.178128 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39877
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:49:49.312108 15987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:49:49.312444 15987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:49:49.312497 15987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:49:49.317497 15987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:49:49.317592 15987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:49:49.317658 15987 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 14:49:49.323742 15987 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39877
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15987
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:49:49.325129 15987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:49:49.326344 15987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:49:49.338148 15992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:49:49.338148 15993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:49:49.338346 15995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:49:49.338451 15987 server_base.cc:1034] running on GCE node
I20260619 14:49:49.339048 15987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:49:49.339774 15987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:49:49.341007 15987 hybrid_clock.cc:648] HybridClock initialized: now 1781880589340957 us; error 51 us; skew 500 ppm
Jun 19 14:49:49 dist-test-slave-2sb1 krb5kdc[15746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880589, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:49:49.344534 15987 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 14:49:49.345724 15987 webserver.cc:460] Webserver started at http://127.0.98.2:35715/ using document root <none> and password file <none>
I20260619 14:49:49.346380 15987 fs_manager.cc:359] Metadata directory not provided
I20260619 14:49:49.346453 15987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:49:49.346697 15987 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:49:49.349103 15987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-1/data/instance:
uuid: "b9dc81733a644dbe85d11ba50f1f4ebc"
format_stamp: "Formatted at 2026-06-19 14:49:49 on dist-test-slave-2sb1"
server_key: "aa672057f3756cee7397d5b34ba3608d"
server_key_iv: "c0db8c6ce83864fa67b387c353a42fbc"
server_key_version: "encryptionkey@0"
I20260619 14:49:49.349766 15987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b9dc81733a644dbe85d11ba50f1f4ebc"
format_stamp: "Formatted at 2026-06-19 14:49:49 on dist-test-slave-2sb1"
server_key: "aa672057f3756cee7397d5b34ba3608d"
server_key_iv: "c0db8c6ce83864fa67b387c353a42fbc"
server_key_version: "encryptionkey@0"
I20260619 14:49:49.354503 15987 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 14:49:49.357317 16002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:49.358826 15987 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:49:49.359023 15987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-1/wal
uuid: "b9dc81733a644dbe85d11ba50f1f4ebc"
format_stamp: "Formatted at 2026-06-19 14:49:49 on dist-test-slave-2sb1"
server_key: "aa672057f3756cee7397d5b34ba3608d"
server_key_iv: "c0db8c6ce83864fa67b387c353a42fbc"
server_key_version: "encryptionkey@0"
I20260619 14:49:49.359167 15987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:49:49.388520 15987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:49:49.392796 15987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:49:49.393033 15987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:49:49.393710 15987 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:49:49.394703 15987 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:49:49.394779 15987 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:49.394855 15987 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:49:49.394892 15987 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:49.407310 15987 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46373
I20260619 14:49:49.407351 16115 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46373 every 8 connection(s)
I20260619 14:49:49.408360 15987 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-1/data/info.pb
I20260619 14:49:49.417264 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 15987
I20260619 14:49:49.417387 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-1/wal/instance
I20260619 14:49:49.417706 392 external_mini_cluster.cc:1428] Setting key 804d0a7dd95f46c459bdff9961894aa7
Jun 19 14:49:49 dist-test-slave-2sb1 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880589, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:49:49.427266 15830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:49.410750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55015 (local address 127.0.98.62:39877)
0619 14:49:49.410931 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:49.410936 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:49.412451 (+ 1515us) server_negotiation.cc:407] Connection header received
0619 14:49:49.413860 (+ 1409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:49.413863 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:49.413921 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:49.414004 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:49:49.415666 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:49.416285 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:49.418279 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:49.418596 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:49.418765 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:49.422372 (+ 3607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:49:49.422376 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:49:49.422378 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:49:49.422406 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:49:49.424474 (+ 2068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:49.425242 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:49.425249 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:49.425254 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:49.425360 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:49.425796 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:49.425803 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:49.425806 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:49.426093 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:49:49.426218 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:49.426904 (+ 686us) server_negotiation.cc:299] Negotiation successful
0619 14:49:49.427061 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 14:49:49.428117 16118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:49.411293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39877 (local address 127.0.98.2:55015)
0619 14:49:49.412180 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:49.412245 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0619 14:49:49.413583 (+ 1338us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:49:49.414282 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:49:49.414290 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:49:49.414822 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:49:49.415391 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:49.415407 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:49.416438 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:49.416441 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:49.417932 (+ 1491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:49.417946 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:49.418800 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:49.418804 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:49.419011 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:49.419740 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:49:49.419763 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:49:49.421973 (+ 2210us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:49:49.424769 (+ 2796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:49:49.424776 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:49:49.424793 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:49:49.425104 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:49:49.425469 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:49:49.425471 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:49:49.425473 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:49:49.425676 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:49:49.426254 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:49:49.426260 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:49:49.426531 (+ 271us) client_negotiation.cc:769] Sending connection context
0619 14:49:49.426788 (+ 257us) client_negotiation.cc:240] Negotiation successful
0619 14:49:49.427075 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"thread_start_us":149,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 14:49:49.429649 16116 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39877
I20260619 14:49:49.429961 16116 heartbeater.cc:464] Registering TS with master...
I20260619 14:49:49.430619 16116 heartbeater.cc:511] Master 127.0.98.62:39877 requested a full tablet report, sending...
I20260619 14:49:49.431995 15791 ts_manager.cc:194] Registered new tserver with Master: b9dc81733a644dbe85d11ba50f1f4ebc (127.0.98.2:46373)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260619 14:49:49.432900 15791 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55015
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 14:49:49.454303 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39877
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:49:49.582062 16121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:49:49.582376 16121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:49:49.582423 16121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:49:49.587018 16121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:49:49.587113 16121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:49:49.587184 16121 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 14:49:49.593243 16121 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39877
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16121
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:49:49.594460 16121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:49:49.595605 16121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:49:49.606475 16126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:49:49.606514 16129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:49:49.606475 16127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:49:49.606839 16121 server_base.cc:1034] running on GCE node
I20260619 14:49:49.607403 16121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:49:49.608121 16121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:49:49.609354 16121 hybrid_clock.cc:648] HybridClock initialized: now 1781880589609253 us; error 117 us; skew 500 ppm
Jun 19 14:49:49 dist-test-slave-2sb1 krb5kdc[15746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880589, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:49:49.612962 16121 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 14:49:49.614071 16121 webserver.cc:460] Webserver started at http://127.0.98.3:44599/ using document root <none> and password file <none>
I20260619 14:49:49.614713 16121 fs_manager.cc:359] Metadata directory not provided
I20260619 14:49:49.614814 16121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:49:49.615067 16121 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:49:49.617458 16121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-2/data/instance:
uuid: "c25b2be704c74c89a191aa1977240d13"
format_stamp: "Formatted at 2026-06-19 14:49:49 on dist-test-slave-2sb1"
server_key: "a78f727aa1c3c1a6f0023647516cf87a"
server_key_iv: "809fe72fe93ce805a089cdda04a1fc8c"
server_key_version: "encryptionkey@0"
I20260619 14:49:49.618108 16121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c25b2be704c74c89a191aa1977240d13"
format_stamp: "Formatted at 2026-06-19 14:49:49 on dist-test-slave-2sb1"
server_key: "a78f727aa1c3c1a6f0023647516cf87a"
server_key_iv: "809fe72fe93ce805a089cdda04a1fc8c"
server_key_version: "encryptionkey@0"
I20260619 14:49:49.622229 16121 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 14:49:49.624979 16136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:49.626626 16121 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 14:49:49.626902 16121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-2/wal
uuid: "c25b2be704c74c89a191aa1977240d13"
format_stamp: "Formatted at 2026-06-19 14:49:49 on dist-test-slave-2sb1"
server_key: "a78f727aa1c3c1a6f0023647516cf87a"
server_key_iv: "809fe72fe93ce805a089cdda04a1fc8c"
server_key_version: "encryptionkey@0"
I20260619 14:49:49.627043 16121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:49:49.647853 16121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:49:49.651823 16121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:49:49.652061 16121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:49:49.652747 16121 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:49:49.653654 16121 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:49:49.653729 16121 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:49.653806 16121 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:49:49.653851 16121 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:49:49.665038 16121 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39361
I20260619 14:49:49.665222 16249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39361 every 8 connection(s)
I20260619 14:49:49.666349 16121 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-2/data/info.pb
I20260619 14:49:49.672470 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 16121
I20260619 14:49:49.672587 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-2/wal/instance
I20260619 14:49:49.672911 392 external_mini_cluster.cc:1428] Setting key 8da558508be9eb8cda281c6d7b46d250
Jun 19 14:49:49 dist-test-slave-2sb1 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880589, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:49:49.684507 15830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:49.668739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58185 (local address 127.0.98.62:39877)
0619 14:49:49.668915 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:49.668920 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:49.669861 (+ 941us) server_negotiation.cc:407] Connection header received
0619 14:49:49.671442 (+ 1581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:49.671445 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:49.671511 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:49.671633 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:49:49.673288 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:49.674365 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:49.675985 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:49.676361 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:49.676467 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:49.679892 (+ 3425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:49:49.679896 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:49:49.679898 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:49:49.679931 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:49:49.681859 (+ 1928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:49.682651 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:49.682654 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:49.682655 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:49.682749 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:49.683152 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:49.683155 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:49.683156 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:49.683361 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:49:49.683448 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:49.684093 (+ 645us) server_negotiation.cc:299] Negotiation successful
0619 14:49:49.684341 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 14:49:49.685441 16252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:49.669142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39877 (local address 127.0.98.3:58185)
0619 14:49:49.669687 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:49.669734 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 14:49:49.671202 (+ 1468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:49:49.671829 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:49:49.671837 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:49:49.672399 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:49:49.673108 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:49.673133 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:49.674586 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:49.674589 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:49.675825 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:49.675838 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:49.676572 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:49.676576 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:49.676857 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:49.677672 (+ 815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:49:49.677691 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:49:49.679641 (+ 1950us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:49:49.682045 (+ 2404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:49:49.682052 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:49:49.682069 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:49:49.682430 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:49:49.682857 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:49:49.682860 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:49:49.682861 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:49:49.683017 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:49:49.683473 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:49:49.683479 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:49:49.683792 (+ 313us) client_negotiation.cc:769] Sending connection context
0619 14:49:49.683967 (+ 175us) client_negotiation.cc:240] Negotiation successful
0619 14:49:49.684413 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":147,"threads_started":1}
I20260619 14:49:49.687192 16250 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39877
I20260619 14:49:49.687506 16250 heartbeater.cc:464] Registering TS with master...
I20260619 14:49:49.688155 16250 heartbeater.cc:511] Master 127.0.98.62:39877 requested a full tablet report, sending...
I20260619 14:49:49.689452 15790 ts_manager.cc:194] Registered new tserver with Master: c25b2be704c74c89a191aa1977240d13 (127.0.98.3:39361)
I20260619 14:49:49.690122 15790 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:58185
I20260619 14:49:49.701105 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:49:49.712707 15830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:49.703442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35056 (local address 127.0.98.62:39877)
0619 14:49:49.703754 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:49.703760 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:49.703905 (+ 145us) server_negotiation.cc:407] Connection header received
0619 14:49:49.704177 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:49.704180 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:49.704227 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:49.704324 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:49:49.705167 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:49.705706 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:49.706641 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:49.706866 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:49.706988 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:49.708730 (+ 1742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:49:49.708734 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:49:49.708736 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:49:49.708764 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:49:49.710595 (+ 1831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:49.711285 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:49.711289 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:49.711290 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:49.711379 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:49:49.711723 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:49:49.711726 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:49:49.711727 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:49:49.711995 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:49:49.712112 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:49.712394 (+ 282us) server_negotiation.cc:299] Negotiation successful
0619 14:49:49.712510 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205}
I20260619 14:49:49.716813 15790 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35056:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:49:49.719916 15790 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:49:49.737841 16261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:49.731168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46373 (local address 127.0.0.1:43664)
0619 14:49:49.731902 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:49.731961 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 14:49:49.732102 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:49:49.733069 (+ 967us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:49:49.733072 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:49:49.733099 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:49:49.733282 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:49.733291 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:49.735111 (+ 1820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:49.735117 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:49.736028 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:49.736038 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:49.737158 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:49.737162 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:49.737301 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:49.737362 (+ 61us) client_negotiation.cc:769] Sending connection context
0619 14:49:49.737500 (+ 138us) client_negotiation.cc:240] Negotiation successful
0619 14:49:49.737651 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"mutex_wait_us":55,"spinlock_wait_cycles":20608,"thread_start_us":121,"threads_started":1}
I20260619 14:49:49.738200 16266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:49.731790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43664 (local address 127.0.98.2:46373)
0619 14:49:49.732570 (+ 780us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:49.732579 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:49.732596 (+ 17us) server_negotiation.cc:407] Connection header received
0619 14:49:49.732721 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:49.732725 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:49.732902 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:49.733088 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:49:49.733901 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:49.734952 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:49.736166 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:49.736991 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:49.737110 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:49.737271 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:49.737928 (+ 657us) server_negotiation.cc:299] Negotiation successful
0619 14:49:49.738036 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":699,"thread_start_us":129,"threads_started":1}
I20260619 14:49:49.738567 16265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:49.730852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39361 (local address 127.0.0.1:54162)
0619 14:49:49.731902 (+ 1050us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:49.731974 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0619 14:49:49.732102 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:49:49.733069 (+ 967us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:49:49.733072 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:49:49.733101 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:49:49.733296 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:49.733301 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:49.734737 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:49.734739 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:49.736361 (+ 1622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:49.736373 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:49.737901 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:49.737905 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:49.738220 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:49.738238 (+ 18us) client_negotiation.cc:769] Sending connection context
0619 14:49:49.738344 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 14:49:49.738419 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":882,"mutex_wait_us":122,"spinlock_wait_cycles":18944,"thread_start_us":99,"threads_started":1}
I20260619 14:49:49.739123 16264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:49.731135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54162 (local address 127.0.98.3:39361)
0619 14:49:49.732428 (+ 1293us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:49.732439 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:49.732458 (+ 19us) server_negotiation.cc:407] Connection header received
0619 14:49:49.732531 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:49.732535 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:49.732722 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:49.732885 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:49:49.733414 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:49.734602 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:49.736812 (+ 2210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:49.737599 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:49.737703 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:49.737835 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:49.738841 (+ 1006us) server_negotiation.cc:299] Negotiation successful
0619 14:49:49.738976 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1188,"thread_start_us":175,"threads_started":1}
I20260619 14:49:49.739163 16263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:49.731122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32768 (local address 127.0.98.1:43063)
0619 14:49:49.731693 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:49.731701 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:49.732348 (+ 647us) server_negotiation.cc:407] Connection header received
0619 14:49:49.732459 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:49.732465 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:49.732725 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:49.732884 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:49:49.733239 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:49.734545 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:49.736523 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:49.737391 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:49.737536 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:49.737685 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:49.737743 (+ 58us) server_negotiation.cc:299] Negotiation successful
0619 14:49:49.739015 (+ 1272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":151,"threads_started":1}
I20260619 14:49:49.739213 16262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:49.730851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43063 (local address 127.0.0.1:32768)
0619 14:49:49.732224 (+ 1373us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:49.732237 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:49:49.732341 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:49:49.732892 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:49:49.732898 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:49:49.732922 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:49:49.733092 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:49.733100 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:49.734724 (+ 1624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:49.734728 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:49.736378 (+ 1650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:49.736385 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:49.737528 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:49.737532 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:49.737626 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:49.737649 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 14:49:49.737715 (+ 66us) client_negotiation.cc:240] Negotiation successful
0619 14:49:49.737792 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1293,"mutex_wait_us":769,"thread_start_us":101,"threads_started":1}
I20260619 14:49:49.741658 16050 tablet_service.cc:1467] Processing CreateTablet for tablet efe8a198d2ea4c39bdd1a51b15bf4c21 (DEFAULT_TABLE table=test-table [id=b8aa58f2a42842398bcd4020d7a5ae64]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:49:49.741958 15916 tablet_service.cc:1467] Processing CreateTablet for tablet efe8a198d2ea4c39bdd1a51b15bf4c21 (DEFAULT_TABLE table=test-table [id=b8aa58f2a42842398bcd4020d7a5ae64]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:49:49.741935 16184 tablet_service.cc:1467] Processing CreateTablet for tablet efe8a198d2ea4c39bdd1a51b15bf4c21 (DEFAULT_TABLE table=test-table [id=b8aa58f2a42842398bcd4020d7a5ae64]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:49:49.742921 16050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efe8a198d2ea4c39bdd1a51b15bf4c21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:49:49.742921 15916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efe8a198d2ea4c39bdd1a51b15bf4c21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:49:49.742933 16184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efe8a198d2ea4c39bdd1a51b15bf4c21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:49:49.749943 16267 tablet_bootstrap.cc:492] T efe8a198d2ea4c39bdd1a51b15bf4c21 P c25b2be704c74c89a191aa1977240d13: Bootstrap starting.
I20260619 14:49:49.751161 16269 tablet_bootstrap.cc:492] T efe8a198d2ea4c39bdd1a51b15bf4c21 P b9dc81733a644dbe85d11ba50f1f4ebc: Bootstrap starting.
I20260619 14:49:49.752975 16267 tablet_bootstrap.cc:654] T efe8a198d2ea4c39bdd1a51b15bf4c21 P c25b2be704c74c89a191aa1977240d13: Neither blocks nor log segments found. Creating new log.
I20260619 14:49:49.753098 16268 tablet_bootstrap.cc:492] T efe8a198d2ea4c39bdd1a51b15bf4c21 P 4bf51bda973949fcb3568cb66538fcdd: Bootstrap starting.
I20260619 14:49:49.753496 16269 tablet_bootstrap.cc:654] T efe8a198d2ea4c39bdd1a51b15bf4c21 P b9dc81733a644dbe85d11ba50f1f4ebc: Neither blocks nor log segments found. Creating new log.
I20260619 14:49:49.754460 16269 log.cc:824] T efe8a198d2ea4c39bdd1a51b15bf4c21 P b9dc81733a644dbe85d11ba50f1f4ebc: Log is configured to *not* fsync() on all Append() calls
I20260619 14:49:49.754560 16267 log.cc:824] T efe8a198d2ea4c39bdd1a51b15bf4c21 P c25b2be704c74c89a191aa1977240d13: Log is configured to *not* fsync() on all Append() calls
I20260619 14:49:49.755538 16268 tablet_bootstrap.cc:654] T efe8a198d2ea4c39bdd1a51b15bf4c21 P 4bf51bda973949fcb3568cb66538fcdd: Neither blocks nor log segments found. Creating new log.
I20260619 14:49:49.756397 16268 log.cc:824] T efe8a198d2ea4c39bdd1a51b15bf4c21 P 4bf51bda973949fcb3568cb66538fcdd: Log is configured to *not* fsync() on all Append() calls
I20260619 14:49:49.756767 16269 tablet_bootstrap.cc:492] T efe8a198d2ea4c39bdd1a51b15bf4c21 P b9dc81733a644dbe85d11ba50f1f4ebc: No bootstrap required, opened a new log
I20260619 14:49:49.756767 16267 tablet_bootstrap.cc:492] T efe8a198d2ea4c39bdd1a51b15bf4c21 P c25b2be704c74c89a191aa1977240d13: No bootstrap required, opened a new log
I20260619 14:49:49.756969 16269 ts_tablet_manager.cc:1397] T efe8a198d2ea4c39bdd1a51b15bf4c21 P b9dc81733a644dbe85d11ba50f1f4ebc: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 14:49:49.756982 16267 ts_tablet_manager.cc:1397] T efe8a198d2ea4c39bdd1a51b15bf4c21 P c25b2be704c74c89a191aa1977240d13: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260619 14:49:49.758618 16268 tablet_bootstrap.cc:492] T efe8a198d2ea4c39bdd1a51b15bf4c21 P 4bf51bda973949fcb3568cb66538fcdd: No bootstrap required, opened a new log
I20260619 14:49:49.758828 16268 ts_tablet_manager.cc:1397] T efe8a198d2ea4c39bdd1a51b15bf4c21 P 4bf51bda973949fcb3568cb66538fcdd: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 14:49:49.760211 16267 raft_consensus.cc:348] T efe8a198d2ea4c39bdd1a51b15bf4c21 P c25b2be704c74c89a191aa1977240d13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bf51bda973949fcb3568cb66538fcdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43063 } } peers { permanent_uuid: "c25b2be704c74c89a191aa1977240d13" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39361 } } peers { permanent_uuid: "b9dc81733a644dbe85d11ba50f1f4ebc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46373 } }
I20260619 14:49:49.760488 16267 raft_consensus.cc:374] T efe8a198d2ea4c39bdd1a51b15bf4c21 P c25b2be704c74c89a191aa1977240d13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:49:49.760555 16267 raft_consensus.cc:729] T efe8a198d2ea4c39bdd1a51b15bf4c21 P c25b2be704c74c89a191aa1977240d13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c25b2be704c74c89a191aa1977240d13, State: Initialized, Role: FOLLOWER
I20260619 14:49:49.761090 16267 consensus_queue.cc:260] T efe8a198d2ea4c39bdd1a51b15bf4c21 P c25b2be704c74c89a191aa1977240d13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bf51bda973949fcb3568cb66538fcdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43063 } } peers { permanent_uuid: "c25b2be704c74c89a191aa1977240d13" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39361 } } peers { permanent_uuid: "b9dc81733a644dbe85d11ba50f1f4ebc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46373 } }
I20260619 14:49:49.761103 16269 raft_consensus.cc:348] T efe8a198d2ea4c39bdd1a51b15bf4c21 P b9dc81733a644dbe85d11ba50f1f4ebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bf51bda973949fcb3568cb66538fcdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43063 } } peers { permanent_uuid: "c25b2be704c74c89a191aa1977240d13" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39361 } } peers { permanent_uuid: "b9dc81733a644dbe85d11ba50f1f4ebc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46373 } }
I20260619 14:49:49.761386 16269 raft_consensus.cc:374] T efe8a198d2ea4c39bdd1a51b15bf4c21 P b9dc81733a644dbe85d11ba50f1f4ebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:49:49.761436 16269 raft_consensus.cc:729] T efe8a198d2ea4c39bdd1a51b15bf4c21 P b9dc81733a644dbe85d11ba50f1f4ebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9dc81733a644dbe85d11ba50f1f4ebc, State: Initialized, Role: FOLLOWER
I20260619 14:49:49.761901 16269 consensus_queue.cc:260] T efe8a198d2ea4c39bdd1a51b15bf4c21 P b9dc81733a644dbe85d11ba50f1f4ebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bf51bda973949fcb3568cb66538fcdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43063 } } peers { permanent_uuid: "c25b2be704c74c89a191aa1977240d13" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39361 } } peers { permanent_uuid: "b9dc81733a644dbe85d11ba50f1f4ebc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46373 } }
I20260619 14:49:49.762207 16267 ts_tablet_manager.cc:1428] T efe8a198d2ea4c39bdd1a51b15bf4c21 P c25b2be704c74c89a191aa1977240d13: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260619 14:49:49.762609 16250 heartbeater.cc:503] Master 127.0.98.62:39877 was elected leader, sending a full tablet report...
I20260619 14:49:49.762961 16269 ts_tablet_manager.cc:1428] T efe8a198d2ea4c39bdd1a51b15bf4c21 P b9dc81733a644dbe85d11ba50f1f4ebc: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 14:49:49.763345 16116 heartbeater.cc:503] Master 127.0.98.62:39877 was elected leader, sending a full tablet report...
I20260619 14:49:49.764806 16268 raft_consensus.cc:348] T efe8a198d2ea4c39bdd1a51b15bf4c21 P 4bf51bda973949fcb3568cb66538fcdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bf51bda973949fcb3568cb66538fcdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43063 } } peers { permanent_uuid: "c25b2be704c74c89a191aa1977240d13" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39361 } } peers { permanent_uuid: "b9dc81733a644dbe85d11ba50f1f4ebc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46373 } }
I20260619 14:49:49.765138 16268 raft_consensus.cc:374] T efe8a198d2ea4c39bdd1a51b15bf4c21 P 4bf51bda973949fcb3568cb66538fcdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:49:49.765213 16268 raft_consensus.cc:729] T efe8a198d2ea4c39bdd1a51b15bf4c21 P 4bf51bda973949fcb3568cb66538fcdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bf51bda973949fcb3568cb66538fcdd, State: Initialized, Role: FOLLOWER
I20260619 14:49:49.765913 16268 consensus_queue.cc:260] T efe8a198d2ea4c39bdd1a51b15bf4c21 P 4bf51bda973949fcb3568cb66538fcdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bf51bda973949fcb3568cb66538fcdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43063 } } peers { permanent_uuid: "c25b2be704c74c89a191aa1977240d13" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39361 } } peers { permanent_uuid: "b9dc81733a644dbe85d11ba50f1f4ebc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46373 } }
I20260619 14:49:49.767284 16268 ts_tablet_manager.cc:1428] T efe8a198d2ea4c39bdd1a51b15bf4c21 P 4bf51bda973949fcb3568cb66538fcdd: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260619 14:49:49.767699 15982 heartbeater.cc:503] Master 127.0.98.62:39877 was elected leader, sending a full tablet report...
I20260619 14:49:49.858657 16273 raft_consensus.cc:482] T efe8a198d2ea4c39bdd1a51b15bf4c21 P c25b2be704c74c89a191aa1977240d13 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:49:49.858888 16273 raft_consensus.cc:504] T efe8a198d2ea4c39bdd1a51b15bf4c21 P c25b2be704c74c89a191aa1977240d13 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bf51bda973949fcb3568cb66538fcdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43063 } } peers { permanent_uuid: "c25b2be704c74c89a191aa1977240d13" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39361 } } peers { permanent_uuid: "b9dc81733a644dbe85d11ba50f1f4ebc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46373 } }
I20260619 14:49:49.860534 16273 leader_election.cc:288] T efe8a198d2ea4c39bdd1a51b15bf4c21 P c25b2be704c74c89a191aa1977240d13 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4bf51bda973949fcb3568cb66538fcdd (127.0.98.1:43063), b9dc81733a644dbe85d11ba50f1f4ebc (127.0.98.2:46373)
I20260619 14:49:49.860908 16275 raft_consensus.cc:482] T efe8a198d2ea4c39bdd1a51b15bf4c21 P 4bf51bda973949fcb3568cb66538fcdd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:49:49.861158 16275 raft_consensus.cc:504] T efe8a198d2ea4c39bdd1a51b15bf4c21 P 4bf51bda973949fcb3568cb66538fcdd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bf51bda973949fcb3568cb66538fcdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43063 } } peers { permanent_uuid: "c25b2be704c74c89a191aa1977240d13" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39361 } } peers { permanent_uuid: "b9dc81733a644dbe85d11ba50f1f4ebc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46373 } }
I20260619 14:49:49.863209 16275 leader_election.cc:288] T efe8a198d2ea4c39bdd1a51b15bf4c21 P 4bf51bda973949fcb3568cb66538fcdd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c25b2be704c74c89a191aa1977240d13 (127.0.98.3:39361), b9dc81733a644dbe85d11ba50f1f4ebc (127.0.98.2:46373)
I20260619 14:49:49.865522 16252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:49.860749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43063 (local address 127.0.98.3:51303)
0619 14:49:49.860956 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:49.860973 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 14:49:49.861092 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:49:49.861525 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:49:49.861528 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:49:49.861548 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:49:49.861681 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:49.861691 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:49.862702 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:49.862705 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:49.863986 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:49.863994 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:49.865074 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:49.865078 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:49.865193 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:49.865221 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 14:49:49.865290 (+ 69us) client_negotiation.cc:240] Negotiation successful
0619 14:49:49.865368 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"mutex_wait_us":1}
I20260619 14:49:49.866039 16263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:49.861020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51303 (local address 127.0.98.1:43063)
0619 14:49:49.861254 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:49.861260 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:49.861276 (+ 16us) server_negotiation.cc:407] Connection header received
0619 14:49:49.861332 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:49.861335 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:49.861389 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:49.861471 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:49:49.861847 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:49.862568 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:49.864108 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:49.864810 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:49.864906 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:49.864983 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:49.865828 (+ 845us) server_negotiation.cc:299] Negotiation successful
0619 14:49:49.865907 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150}
I20260619 14:49:49.866251 16266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:49.861294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44095 (local address 127.0.98.2:46373)
0619 14:49:49.861446 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:49.861453 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:49.861541 (+ 88us) server_negotiation.cc:407] Connection header received
0619 14:49:49.861784 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:49.861791 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:49.861862 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:49.861962 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:49:49.862244 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:49.863030 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:49.864897 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:49.865497 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:49.865591 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:49.865677 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:49.866041 (+ 364us) server_negotiation.cc:299] Negotiation successful
0619 14:49:49.866126 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 14:49:49.866655 16276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:49.860874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46373 (local address 127.0.98.3:44095)
0619 14:49:49.861444 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:49.861459 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:49:49.861616 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:49:49.861969 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:49:49.861972 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:49:49.861989 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:49:49.862125 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:49.862130 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:49.863136 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:49.863143 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:49.864740 (+ 1597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:49.864753 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:49.865790 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:49.865793 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:49.865899 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:49.865922 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 14:49:49.866446 (+ 524us) client_negotiation.cc:240] Negotiation successful
0619 14:49:49.866522 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":255,"threads_started":1}
I20260619 14:49:49.866657 15936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efe8a198d2ea4c39bdd1a51b15bf4c21" candidate_uuid: "c25b2be704c74c89a191aa1977240d13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bf51bda973949fcb3568cb66538fcdd" is_pre_election: true
I20260619 14:49:49.866904 15936 raft_consensus.cc:2454] T efe8a198d2ea4c39bdd1a51b15bf4c21 P 4bf51bda973949fcb3568cb66538fcdd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c25b2be704c74c89a191aa1977240d13 in term 0.
I20260619 14:49:49.867379 16070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efe8a198d2ea4c39bdd1a51b15bf4c21" candidate_uuid: "c25b2be704c74c89a191aa1977240d13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9dc81733a644dbe85d11ba50f1f4ebc" is_pre_election: true
I20260619 14:49:49.867431 16137 leader_election.cc:302] T efe8a198d2ea4c39bdd1a51b15bf4c21 P c25b2be704c74c89a191aa1977240d13 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bf51bda973949fcb3568cb66538fcdd, c25b2be704c74c89a191aa1977240d13; no voters:
I20260619 14:49:49.867700 16070 raft_consensus.cc:2454] T efe8a198d2ea4c39bdd1a51b15bf4c21 P b9dc81733a644dbe85d11ba50f1f4ebc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c25b2be704c74c89a191aa1977240d13 in term 0.
I20260619 14:49:49.867779 16273 raft_consensus.cc:2781] T efe8a198d2ea4c39bdd1a51b15bf4c21 P c25b2be704c74c89a191aa1977240d13 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:49:49.867873 16273 raft_consensus.cc:482] T efe8a198d2ea4c39bdd1a51b15bf4c21 P c25b2be704c74c89a191aa1977240d13 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:49:49.867919 16273 raft_consensus.cc:3037] T efe8a198d2ea4c39bdd1a51b15bf4c21 P c25b2be704c74c89a191aa1977240d13 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:49:49.868368 16278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:49.862714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46373 (local address 127.0.98.1:41405)
0619 14:49:49.863784 (+ 1070us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:49.863796 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:49:49.863869 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:49:49.864212 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:49:49.864215 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:49:49.864285 (+ 70us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:49:49.864400 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:49.864406 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:49.865668 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:49.865671 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:49.866903 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:49.866914 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:49.867895 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:49.867898 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:49.868014 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:49.868043 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 14:49:49.868111 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 14:49:49.868202 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1012,"thread_start_us":136,"threads_started":1}
I20260619 14:49:49.868686 16264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:49.863472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33013 (local address 127.0.98.3:39361)
0619 14:49:49.863734 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:49.863738 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:49.863749 (+ 11us) server_negotiation.cc:407] Connection header received
0619 14:49:49.863788 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:49.863790 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:49.863826 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:49.863897 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:49:49.864295 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:49.865024 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:49.867218 (+ 2194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:49.867776 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:49.867862 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:49.867931 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:49.868463 (+ 532us) server_negotiation.cc:299] Negotiation successful
0619 14:49:49.868544 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202}
I20260619 14:49:49.868765 16279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:49.863149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41405 (local address 127.0.98.2:46373)
0619 14:49:49.863634 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:49.863648 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:49.863946 (+ 298us) server_negotiation.cc:407] Connection header received
0619 14:49:49.863994 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:49.863998 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:49.864041 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:49.864116 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:49:49.864610 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:49.865551 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:49.867035 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:49.867657 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:49.867769 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:49.867828 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:49.868533 (+ 705us) server_negotiation.cc:299] Negotiation successful
0619 14:49:49.868608 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":72,"threads_started":1}
I20260619 14:49:49.869216 16277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:49.862698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39361 (local address 127.0.98.1:33013)
0619 14:49:49.863470 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:49.863486 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:49:49.863601 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:49:49.864006 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:49:49.864009 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:49:49.864033 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:49:49.864169 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:49.864175 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:49.865154 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:49.865158 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:49.867074 (+ 1916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:49:49.867083 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:49:49.868192 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:49:49.868199 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:49:49.868305 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:49.868329 (+ 24us) client_negotiation.cc:769] Sending connection context
0619 14:49:49.868410 (+ 81us) client_negotiation.cc:240] Negotiation successful
0619 14:49:49.869069 (+ 659us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":700,"thread_start_us":159,"threads_started":1}
I20260619 14:49:49.869184 16070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efe8a198d2ea4c39bdd1a51b15bf4c21" candidate_uuid: "4bf51bda973949fcb3568cb66538fcdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9dc81733a644dbe85d11ba50f1f4ebc" is_pre_election: true
I20260619 14:49:49.869354 16070 raft_consensus.cc:2454] T efe8a198d2ea4c39bdd1a51b15bf4c21 P b9dc81733a644dbe85d11ba50f1f4ebc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bf51bda973949fcb3568cb66538fcdd in term 0.
I20260619 14:49:49.869820 16204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efe8a198d2ea4c39bdd1a51b15bf4c21" candidate_uuid: "4bf51bda973949fcb3568cb66538fcdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c25b2be704c74c89a191aa1977240d13" is_pre_election: true
I20260619 14:49:49.869845 15872 leader_election.cc:302] T efe8a198d2ea4c39bdd1a51b15bf4c21 P 4bf51bda973949fcb3568cb66538fcdd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bf51bda973949fcb3568cb66538fcdd, b9dc81733a644dbe85d11ba50f1f4ebc; no voters:
I20260619 14:49:49.869964 16273 raft_consensus.cc:504] T efe8a198d2ea4c39bdd1a51b15bf4c21 P c25b2be704c74c89a191aa1977240d13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bf51bda973949fcb3568cb66538fcdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43063 } } peers { permanent_uuid: "c25b2be704c74c89a191aa1977240d13" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39361 } } peers { permanent_uuid: "b9dc81733a644dbe85d11ba50f1f4ebc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46373 } }
I20260619 14:49:49.870159 16275 raft_consensus.cc:2781] T efe8a198d2ea4c39bdd1a51b15bf4c21 P 4bf51bda973949fcb3568cb66538fcdd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:49:49.870266 16275 raft_consensus.cc:482] T efe8a198d2ea4c39bdd1a51b15bf4c21 P 4bf51bda973949fcb3568cb66538fcdd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:49:49.870287 16204 raft_consensus.cc:2379] T efe8a198d2ea4c39bdd1a51b15bf4c21 P c25b2be704c74c89a191aa1977240d13 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4bf51bda973949fcb3568cb66538fcdd in current term 1: Already voted for candidate c25b2be704c74c89a191aa1977240d13 in this term.
I20260619 14:49:49.870323 16275 raft_consensus.cc:3037] T efe8a198d2ea4c39bdd1a51b15bf4c21 P 4bf51bda973949fcb3568cb66538fcdd [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:49:49.870788 15936 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efe8a198d2ea4c39bdd1a51b15bf4c21" candidate_uuid: "c25b2be704c74c89a191aa1977240d13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bf51bda973949fcb3568cb66538fcdd"
I20260619 14:49:49.871222 16273 leader_election.cc:288] T efe8a198d2ea4c39bdd1a51b15bf4c21 P c25b2be704c74c89a191aa1977240d13 [CANDIDATE]: Term 1 election: Requested vote from peers 4bf51bda973949fcb3568cb66538fcdd (127.0.98.1:43063), b9dc81733a644dbe85d11ba50f1f4ebc (127.0.98.2:46373)
I20260619 14:49:49.871305 16070 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efe8a198d2ea4c39bdd1a51b15bf4c21" candidate_uuid: "c25b2be704c74c89a191aa1977240d13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9dc81733a644dbe85d11ba50f1f4ebc"
I20260619 14:49:49.871438 16070 raft_consensus.cc:3037] T efe8a198d2ea4c39bdd1a51b15bf4c21 P b9dc81733a644dbe85d11ba50f1f4ebc [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:49:49.871994 16275 raft_consensus.cc:504] T efe8a198d2ea4c39bdd1a51b15bf4c21 P 4bf51bda973949fcb3568cb66538fcdd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bf51bda973949fcb3568cb66538fcdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43063 } } peers { permanent_uuid: "c25b2be704c74c89a191aa1977240d13" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39361 } } peers { permanent_uuid: "b9dc81733a644dbe85d11ba50f1f4ebc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46373 } }
I20260619 14:49:49.872262 15936 raft_consensus.cc:2379] T efe8a198d2ea4c39bdd1a51b15bf4c21 P 4bf51bda973949fcb3568cb66538fcdd [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate c25b2be704c74c89a191aa1977240d13 in current term 1: Already voted for candidate 4bf51bda973949fcb3568cb66538fcdd in this term.
I20260619 14:49:49.872442 16275 leader_election.cc:288] T efe8a198d2ea4c39bdd1a51b15bf4c21 P 4bf51bda973949fcb3568cb66538fcdd [CANDIDATE]: Term 1 election: Requested vote from peers c25b2be704c74c89a191aa1977240d13 (127.0.98.3:39361), b9dc81733a644dbe85d11ba50f1f4ebc (127.0.98.2:46373)
I20260619 14:49:49.872860 16070 raft_consensus.cc:2454] T efe8a198d2ea4c39bdd1a51b15bf4c21 P b9dc81733a644dbe85d11ba50f1f4ebc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c25b2be704c74c89a191aa1977240d13 in term 1.
I20260619 14:49:49.873217 16069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efe8a198d2ea4c39bdd1a51b15bf4c21" candidate_uuid: "4bf51bda973949fcb3568cb66538fcdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9dc81733a644dbe85d11ba50f1f4ebc"
I20260619 14:49:49.873406 16069 raft_consensus.cc:2379] T efe8a198d2ea4c39bdd1a51b15bf4c21 P b9dc81733a644dbe85d11ba50f1f4ebc [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 4bf51bda973949fcb3568cb66538fcdd in current term 1: Already voted for candidate c25b2be704c74c89a191aa1977240d13 in this term.
I20260619 14:49:49.873464 16140 leader_election.cc:302] T efe8a198d2ea4c39bdd1a51b15bf4c21 P c25b2be704c74c89a191aa1977240d13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b9dc81733a644dbe85d11ba50f1f4ebc, c25b2be704c74c89a191aa1977240d13; no voters:
I20260619 14:49:49.873473 16204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efe8a198d2ea4c39bdd1a51b15bf4c21" candidate_uuid: "4bf51bda973949fcb3568cb66538fcdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c25b2be704c74c89a191aa1977240d13"
I20260619 14:49:49.873685 16273 raft_consensus.cc:2781] T efe8a198d2ea4c39bdd1a51b15bf4c21 P c25b2be704c74c89a191aa1977240d13 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:49:49.873963 16273 raft_consensus.cc:686] T efe8a198d2ea4c39bdd1a51b15bf4c21 P c25b2be704c74c89a191aa1977240d13 [term 1 LEADER]: Becoming Leader. State: Replica: c25b2be704c74c89a191aa1977240d13, State: Running, Role: LEADER
I20260619 14:49:49.874357 16273 consensus_queue.cc:237] T efe8a198d2ea4c39bdd1a51b15bf4c21 P c25b2be704c74c89a191aa1977240d13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bf51bda973949fcb3568cb66538fcdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43063 } } peers { permanent_uuid: "c25b2be704c74c89a191aa1977240d13" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39361 } } peers { permanent_uuid: "b9dc81733a644dbe85d11ba50f1f4ebc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46373 } }
I20260619 14:49:49.877765 15870 leader_election.cc:302] T efe8a198d2ea4c39bdd1a51b15bf4c21 P 4bf51bda973949fcb3568cb66538fcdd [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bf51bda973949fcb3568cb66538fcdd; no voters: b9dc81733a644dbe85d11ba50f1f4ebc, c25b2be704c74c89a191aa1977240d13
I20260619 14:49:49.878031 16275 raft_consensus.cc:2726] T efe8a198d2ea4c39bdd1a51b15bf4c21 P 4bf51bda973949fcb3568cb66538fcdd [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260619 14:49:49.878911 15790 catalog_manager.cc:5495] T efe8a198d2ea4c39bdd1a51b15bf4c21 P c25b2be704c74c89a191aa1977240d13 reported cstate change: term changed from 0 to 1, leader changed from <none> to c25b2be704c74c89a191aa1977240d13 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "c25b2be704c74c89a191aa1977240d13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bf51bda973949fcb3568cb66538fcdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43063 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c25b2be704c74c89a191aa1977240d13" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b9dc81733a644dbe85d11ba50f1f4ebc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46373 } health_report { overall_health: UNKNOWN } } }
W20260619 14:49:49.888532 15983 tablet.cc:2369] T efe8a198d2ea4c39bdd1a51b15bf4c21 P 4bf51bda973949fcb3568cb66538fcdd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:49:49.896554 16264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:49.892438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54170 (local address 127.0.98.3:39361)
0619 14:49:49.892706 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0619 14:49:49.892716 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 14:49:49.892729 (+ 13us) server_negotiation.cc:407] Connection header received
0619 14:49:49.892813 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:49:49.892816 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:49:49.892869 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:49:49.893128 (+ 259us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:49:49.893407 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:49.893925 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:49.894918 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:49:49.895151 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:49:49.895325 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:49:49.895680 (+ 355us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:49:49.896136 (+ 456us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:49:49.896282 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 14:49:49.896328 (+ 46us) server_negotiation.cc:299] Negotiation successful
0619 14:49:49.896388 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129}
I20260619 14:49:49.903162 15936 raft_consensus.cc:1261] T efe8a198d2ea4c39bdd1a51b15bf4c21 P 4bf51bda973949fcb3568cb66538fcdd [term 1 FOLLOWER]: Refusing update from remote peer c25b2be704c74c89a191aa1977240d13: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:49:49.903246 16069 raft_consensus.cc:1261] T efe8a198d2ea4c39bdd1a51b15bf4c21 P b9dc81733a644dbe85d11ba50f1f4ebc [term 1 FOLLOWER]: Refusing update from remote peer c25b2be704c74c89a191aa1977240d13: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:49:49.903889 16273 consensus_queue.cc:1035] T efe8a198d2ea4c39bdd1a51b15bf4c21 P c25b2be704c74c89a191aa1977240d13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bf51bda973949fcb3568cb66538fcdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:49:49.904151 16280 consensus_queue.cc:1035] T efe8a198d2ea4c39bdd1a51b15bf4c21 P c25b2be704c74c89a191aa1977240d13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9dc81733a644dbe85d11ba50f1f4ebc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260619 14:49:49.909632 16117 tablet.cc:2369] T efe8a198d2ea4c39bdd1a51b15bf4c21 P b9dc81733a644dbe85d11ba50f1f4ebc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:49:49.912420 16284 mvcc.cc:205] Tried to move back new op lower bound from 7298582896234393600 to 7298582896128700416. Current Snapshot: MvccSnapshot[applied={T|T < 7298582896234393600}]
I20260619 14:49:49.912631 16283 mvcc.cc:205] Tried to move back new op lower bound from 7298582896234393600 to 7298582896128700416. Current Snapshot: MvccSnapshot[applied={T|T < 7298582896234393600}]
I20260619 14:49:49.921310 16285 mvcc.cc:205] Tried to move back new op lower bound from 7298582896234393600 to 7298582896128700416. Current Snapshot: MvccSnapshot[applied={T|T < 7298582896234393600 or (T in {7298582896234393600})}]
I20260619 14:49:49.922458 15790 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35056:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:49:49.922627 15790 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35056:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:49:49.925393 15790 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 85f59d8a5c854579995d8ebfc3501da5: Sending DeleteTablet for 3 replicas of tablet efe8a198d2ea4c39bdd1a51b15bf4c21
I20260619 14:49:49.926301 16050 tablet_service.cc:1514] Processing DeleteTablet for tablet efe8a198d2ea4c39bdd1a51b15bf4c21 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:49:49 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43664
I20260619 14:49:49.926309 16184 tablet_service.cc:1514] Processing DeleteTablet for tablet efe8a198d2ea4c39bdd1a51b15bf4c21 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:49:49 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54162
I20260619 14:49:49.926309 15916 tablet_service.cc:1514] Processing DeleteTablet for tablet efe8a198d2ea4c39bdd1a51b15bf4c21 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:49:49 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:32768
I20260619 14:49:49.926887 16295 tablet_replica.cc:330] T efe8a198d2ea4c39bdd1a51b15bf4c21 P b9dc81733a644dbe85d11ba50f1f4ebc: stopping tablet replica
I20260619 14:49:49.927078 16294 tablet_replica.cc:330] T efe8a198d2ea4c39bdd1a51b15bf4c21 P c25b2be704c74c89a191aa1977240d13: stopping tablet replica
I20260619 14:49:49.927353 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 15853
I20260619 14:49:49.927351 16295 raft_consensus.cc:2229] T efe8a198d2ea4c39bdd1a51b15bf4c21 P b9dc81733a644dbe85d11ba50f1f4ebc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:49:49.927350 16294 raft_consensus.cc:2229] T efe8a198d2ea4c39bdd1a51b15bf4c21 P c25b2be704c74c89a191aa1977240d13 [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:49:49.927734 16294 raft_consensus.cc:2258] T efe8a198d2ea4c39bdd1a51b15bf4c21 P c25b2be704c74c89a191aa1977240d13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:49:49.929612 16294 ts_tablet_manager.cc:1905] T efe8a198d2ea4c39bdd1a51b15bf4c21 P c25b2be704c74c89a191aa1977240d13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:49:49.930130 16295 raft_consensus.cc:2258] T efe8a198d2ea4c39bdd1a51b15bf4c21 P b9dc81733a644dbe85d11ba50f1f4ebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:49:49.932011 16295 ts_tablet_manager.cc:1905] T efe8a198d2ea4c39bdd1a51b15bf4c21 P b9dc81733a644dbe85d11ba50f1f4ebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:49:49.932710 16294 ts_tablet_manager.cc:1918] T efe8a198d2ea4c39bdd1a51b15bf4c21 P c25b2be704c74c89a191aa1977240d13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:49:49.932814 16294 log.cc:1196] T efe8a198d2ea4c39bdd1a51b15bf4c21 P c25b2be704c74c89a191aa1977240d13: Deleting WAL directory at /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-2/wal/wals/efe8a198d2ea4c39bdd1a51b15bf4c21
I20260619 14:49:49.933168 16294 ts_tablet_manager.cc:1939] T efe8a198d2ea4c39bdd1a51b15bf4c21 P c25b2be704c74c89a191aa1977240d13: Deleting consensus metadata
I20260619 14:49:49.934778 15776 catalog_manager.cc:4841] TS c25b2be704c74c89a191aa1977240d13 (127.0.98.3:39361): tablet efe8a198d2ea4c39bdd1a51b15bf4c21 (table test-table [id=b8aa58f2a42842398bcd4020d7a5ae64]) successfully deleted
I20260619 14:49:49.935353 16295 ts_tablet_manager.cc:1918] T efe8a198d2ea4c39bdd1a51b15bf4c21 P b9dc81733a644dbe85d11ba50f1f4ebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:49:49.935487 16295 log.cc:1196] T efe8a198d2ea4c39bdd1a51b15bf4c21 P b9dc81733a644dbe85d11ba50f1f4ebc: Deleting WAL directory at /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-1/wal/wals/efe8a198d2ea4c39bdd1a51b15bf4c21
I20260619 14:49:49.935806 16295 ts_tablet_manager.cc:1939] T efe8a198d2ea4c39bdd1a51b15bf4c21 P b9dc81733a644dbe85d11ba50f1f4ebc: Deleting consensus metadata
I20260619 14:49:49.937065 15778 catalog_manager.cc:4841] TS b9dc81733a644dbe85d11ba50f1f4ebc (127.0.98.2:46373): tablet efe8a198d2ea4c39bdd1a51b15bf4c21 (table test-table [id=b8aa58f2a42842398bcd4020d7a5ae64]) successfully deleted
W20260619 14:49:49.939898 15775 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43063 (error 108)
I20260619 14:49:49.940832 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 15987
I20260619 14:49:49.941109 16262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:49:49.940774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43063 (local address 127.0.0.1:32776)
0619 14:49:49.940908 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0619 14:49:49.940997 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260619 14:49:49.941968 15775 catalog_manager.cc:4584] TS 4bf51bda973949fcb3568cb66538fcdd (127.0.98.1:43063): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet efe8a198d2ea4c39bdd1a51b15bf4c21: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43063: connect: Connection refused (error 111)
I20260619 14:49:49.947309 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 16121
I20260619 14:49:49.953392 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 15759
I20260619 14:50:09.960935 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39877
--webserver_interface=127.0.98.62
--webserver_port=35415
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39877
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:50:10.094242 16297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:50:10.094630 16297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:50:10.094717 16297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:50:10.099409 16297 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 14:50:10.099519 16297 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 14:50:10.099546 16297 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 14:50:10.099570 16297 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:50:10.099618 16297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:50:10.099684 16297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:50:10.099736 16297 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:50:10.099778 16297 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:50:10.105971 16297 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39877
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=35415
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16297
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:50:10.107484 16297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:50:10.109293 16297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:50:10.125931 16304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:10.125913 16303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:50:10.126116 16297 server_base.cc:1034] running on GCE node
W20260619 14:50:10.125913 16306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:50:10.127158 16297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:50:10.128502 16297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:50:10.129738 16297 hybrid_clock.cc:648] HybridClock initialized: now 1781880610129703 us; error 55 us; skew 500 ppm
Jun 19 14:50:10 dist-test-slave-2sb1 krb5kdc[15746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880610, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:50:10.133858 16297 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 14:50:10.135048 16297 webserver.cc:460] Webserver started at http://127.0.98.62:35415/ using document root <none> and password file <none>
I20260619 14:50:10.135763 16297 fs_manager.cc:359] Metadata directory not provided
I20260619 14:50:10.136101 16297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:50:10.141215 16297 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:50:10.143834 16313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:10.145607 16297 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:50:10.145745 16297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/master-0/wal
uuid: "85f59d8a5c854579995d8ebfc3501da5"
format_stamp: "Formatted at 2026-06-19 14:49:48 on dist-test-slave-2sb1"
server_key: "bdf7dd6b4385ff46418d8542894d08c3"
server_key_iv: "0d22b10314e11f96a24cb8d704875a66"
server_key_version: "encryptionkey@0"
I20260619 14:50:10.146396 16297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:50:10.162025 16297 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:50:10.166172 16297 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:50:10.166365 16297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:50:10.175640 16297 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39877
I20260619 14:50:10.175927 16365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39877 every 8 connection(s)
I20260619 14:50:10.177066 16297 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:50:10.179788 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 16297
I20260619 14:50:10.180361 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:43063
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:39877
--webserver_port=32955
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:50:10.182459 16366 sys_catalog.cc:263] Verifying existing consensus state
I20260619 14:50:10.185432 16366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85f59d8a5c854579995d8ebfc3501da5: Bootstrap starting.
I20260619 14:50:10.213035 16366 log.cc:824] T 00000000000000000000000000000000 P 85f59d8a5c854579995d8ebfc3501da5: Log is configured to *not* fsync() on all Append() calls
I20260619 14:50:10.226042 16366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85f59d8a5c854579995d8ebfc3501da5: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 14:50:10.226569 16366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85f59d8a5c854579995d8ebfc3501da5: Bootstrap complete.
I20260619 14:50:10.231410 16366 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85f59d8a5c854579995d8ebfc3501da5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85f59d8a5c854579995d8ebfc3501da5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39877 } }
I20260619 14:50:10.231906 16366 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85f59d8a5c854579995d8ebfc3501da5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f59d8a5c854579995d8ebfc3501da5, State: Initialized, Role: FOLLOWER
I20260619 14:50:10.232542 16366 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85f59d8a5c854579995d8ebfc3501da5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85f59d8a5c854579995d8ebfc3501da5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39877 } }
I20260619 14:50:10.232743 16366 raft_consensus.cc:388] T 00000000000000000000000000000000 P 85f59d8a5c854579995d8ebfc3501da5 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:50:10.232833 16366 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85f59d8a5c854579995d8ebfc3501da5 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:50:10.232997 16366 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85f59d8a5c854579995d8ebfc3501da5 [term 1 FOLLOWER]: Advancing to term 2
I20260619 14:50:10.235334 16366 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85f59d8a5c854579995d8ebfc3501da5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85f59d8a5c854579995d8ebfc3501da5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39877 } }
I20260619 14:50:10.235766 16366 leader_election.cc:302] T 00000000000000000000000000000000 P 85f59d8a5c854579995d8ebfc3501da5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85f59d8a5c854579995d8ebfc3501da5; no voters:
I20260619 14:50:10.236181 16366 leader_election.cc:288] T 00000000000000000000000000000000 P 85f59d8a5c854579995d8ebfc3501da5 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260619 14:50:10.236354 16370 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 85f59d8a5c854579995d8ebfc3501da5 [term 2 FOLLOWER]: Leader election won for term 2
I20260619 14:50:10.236724 16370 raft_consensus.cc:686] T 00000000000000000000000000000000 P 85f59d8a5c854579995d8ebfc3501da5 [term 2 LEADER]: Becoming Leader. State: Replica: 85f59d8a5c854579995d8ebfc3501da5, State: Running, Role: LEADER
I20260619 14:50:10.237118 16370 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85f59d8a5c854579995d8ebfc3501da5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85f59d8a5c854579995d8ebfc3501da5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39877 } }
I20260619 14:50:10.237700 16366 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85f59d8a5c854579995d8ebfc3501da5 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:50:10.238912 16371 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85f59d8a5c854579995d8ebfc3501da5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "85f59d8a5c854579995d8ebfc3501da5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85f59d8a5c854579995d8ebfc3501da5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39877 } } }
I20260619 14:50:10.238961 16372 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85f59d8a5c854579995d8ebfc3501da5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85f59d8a5c854579995d8ebfc3501da5. Latest consensus state: current_term: 2 leader_uuid: "85f59d8a5c854579995d8ebfc3501da5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85f59d8a5c854579995d8ebfc3501da5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39877 } } }
I20260619 14:50:10.239068 16371 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85f59d8a5c854579995d8ebfc3501da5 [sys.catalog]: This master's current role is: LEADER
I20260619 14:50:10.239079 16372 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85f59d8a5c854579995d8ebfc3501da5 [sys.catalog]: This master's current role is: LEADER
I20260619 14:50:10.239485 16379 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:50:10.244923 16379 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:50:10.245699 16379 catalog_manager.cc:1259] Loaded cluster ID: 3837edbdb3314120983e94b9eea509df
I20260619 14:50:10.245812 16379 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:50:10.249003 16379 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:50:10.249967 16379 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 85f59d8a5c854579995d8ebfc3501da5: Loaded TSK: 0
I20260619 14:50:10.271770 16379 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 85f59d8a5c854579995d8ebfc3501da5: Generated new TSK 1
I20260619 14:50:10.272152 16379 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 14:50:10.336827 16368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:50:10.337131 16368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:50:10.337175 16368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:50:10.341944 16368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:50:10.342026 16368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:50:10.342092 16368 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 14:50:10.347944 16368 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:43063
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=32955
--tserver_master_addrs=127.0.98.62:39877
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16368
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:50:10.349370 16368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:50:10.350666 16368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:50:10.362571 16394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:10.362453 16396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:10.362453 16393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:50:10.363085 16368 server_base.cc:1034] running on GCE node
I20260619 14:50:10.363615 16368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:50:10.364529 16368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:50:10.365753 16368 hybrid_clock.cc:648] HybridClock initialized: now 1781880610365704 us; error 64 us; skew 500 ppm
Jun 19 14:50:10 dist-test-slave-2sb1 krb5kdc[15746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880610, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:50:10.369987 16368 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 14:50:10.371344 16368 webserver.cc:460] Webserver started at http://127.0.98.1:32955/ using document root <none> and password file <none>
I20260619 14:50:10.372164 16368 fs_manager.cc:359] Metadata directory not provided
I20260619 14:50:10.372288 16368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:50:10.378583 16368 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 14:50:10.381989 16403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:10.383734 16368 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:50:10.383894 16368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-0/wal
uuid: "4bf51bda973949fcb3568cb66538fcdd"
format_stamp: "Formatted at 2026-06-19 14:49:49 on dist-test-slave-2sb1"
server_key: "abb9f876c53f30d1f950e3f76a32d324"
server_key_iv: "f4a116fee231edd0276c783399b9b061"
server_key_version: "encryptionkey@0"
I20260619 14:50:10.384711 16368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:50:10.409464 16368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:50:10.414301 16368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:50:10.414548 16368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:50:10.415345 16368 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:50:10.416836 16410 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 14:50:10.421821 16368 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 14:50:10.421957 16368 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260619 14:50:10.422031 16368 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 14:50:10.424027 16368 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 14:50:10.424089 16368 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260619 14:50:10.424547 16410 tablet_bootstrap.cc:492] T efe8a198d2ea4c39bdd1a51b15bf4c21 P 4bf51bda973949fcb3568cb66538fcdd: Bootstrap starting.
I20260619 14:50:10.437023 16368 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43063
I20260619 14:50:10.437131 16517 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43063 every 8 connection(s)
I20260619 14:50:10.438270 16368 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-0/data/info.pb
I20260619 14:50:10.440779 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 16368
I20260619 14:50:10.441329 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:46373
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:39877
--webserver_port=35715
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:50:10.449220 16410 log.cc:824] T efe8a198d2ea4c39bdd1a51b15bf4c21 P 4bf51bda973949fcb3568cb66538fcdd: Log is configured to *not* fsync() on all Append() calls
Jun 19 14:50:10 dist-test-slave-2sb1 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880610, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:50:10.460243 16410 tablet_bootstrap.cc:492] T efe8a198d2ea4c39bdd1a51b15bf4c21 P 4bf51bda973949fcb3568cb66538fcdd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 14:50:10.461241 16410 tablet_bootstrap.cc:492] T efe8a198d2ea4c39bdd1a51b15bf4c21 P 4bf51bda973949fcb3568cb66538fcdd: Bootstrap complete.
I20260619 14:50:10.462883 16410 ts_tablet_manager.cc:1397] T efe8a198d2ea4c39bdd1a51b15bf4c21 P 4bf51bda973949fcb3568cb66538fcdd: Time spent bootstrapping tablet: real 0.039s user 0.023s sys 0.012s
I20260619 14:50:10.464937 16520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:10.441368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49329 (local address 127.0.98.62:39877)
0619 14:50:10.442331 (+ 963us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:10.442354 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:10.442914 (+ 560us) server_negotiation.cc:407] Connection header received
0619 14:50:10.444949 (+ 2035us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:10.444963 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:10.445333 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:10.445664 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:10.447617 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:10.449200 (+ 1583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:10.451844 (+ 2644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:10.452616 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:10.452860 (+ 244us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:10.456617 (+ 3757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:10.456625 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:10.456668 (+ 43us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:10.456712 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:10.460204 (+ 3492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:10.461283 (+ 1079us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:10.461294 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:10.461301 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:10.461510 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:10.461988 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:10.461995 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:10.462000 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:10.462582 (+ 582us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:10.462753 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:10.463616 (+ 863us) server_negotiation.cc:299] Negotiation successful
0619 14:50:10.463980 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":739,"thread_start_us":149,"threads_started":1}
I20260619 14:50:10.464951 16521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:10.441450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39877 (local address 127.0.98.1:49329)
0619 14:50:10.442697 (+ 1247us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:10.442766 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:10.444275 (+ 1509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:10.445849 (+ 1574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:10.445859 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:10.446491 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:10.447328 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:10.447358 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:10.449537 (+ 2179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:10.449544 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:10.451645 (+ 2101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:10.451667 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:10.452856 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:10.452863 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:10.453144 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:10.453989 (+ 845us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:10.454013 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:10.456284 (+ 2271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:50:10.460509 (+ 4225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:10.460518 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:10.460535 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:10.461089 (+ 554us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:10.461648 (+ 559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:10.461652 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:10.461654 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:10.461864 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:10.462770 (+ 906us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:50:10.462798 (+ 28us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:50:10.463172 (+ 374us) client_negotiation.cc:769] Sending connection context
0619 14:50:10.463445 (+ 273us) client_negotiation.cc:240] Negotiation successful
0619 14:50:10.463842 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":997,"thread_start_us":137,"threads_started":1}
I20260619 14:50:10.466324 16410 raft_consensus.cc:348] T efe8a198d2ea4c39bdd1a51b15bf4c21 P 4bf51bda973949fcb3568cb66538fcdd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bf51bda973949fcb3568cb66538fcdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43063 } } peers { permanent_uuid: "c25b2be704c74c89a191aa1977240d13" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39361 } } peers { permanent_uuid: "b9dc81733a644dbe85d11ba50f1f4ebc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46373 } }
I20260619 14:50:10.466936 16410 raft_consensus.cc:729] T efe8a198d2ea4c39bdd1a51b15bf4c21 P 4bf51bda973949fcb3568cb66538fcdd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bf51bda973949fcb3568cb66538fcdd, State: Initialized, Role: FOLLOWER
I20260619 14:50:10.467550 16410 consensus_queue.cc:260] T efe8a198d2ea4c39bdd1a51b15bf4c21 P 4bf51bda973949fcb3568cb66538fcdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bf51bda973949fcb3568cb66538fcdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43063 } } peers { permanent_uuid: "c25b2be704c74c89a191aa1977240d13" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39361 } } peers { permanent_uuid: "b9dc81733a644dbe85d11ba50f1f4ebc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46373 } }
I20260619 14:50:10.468741 16410 ts_tablet_manager.cc:1428] T efe8a198d2ea4c39bdd1a51b15bf4c21 P 4bf51bda973949fcb3568cb66538fcdd: Time spent starting tablet: real 0.006s user 0.007s sys 0.001s
I20260619 14:50:10.468804 16518 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39877
I20260619 14:50:10.469050 16518 heartbeater.cc:464] Registering TS with master...
I20260619 14:50:10.469609 16518 heartbeater.cc:511] Master 127.0.98.62:39877 requested a full tablet report, sending...
I20260619 14:50:10.472620 16324 ts_manager.cc:194] Registered new tserver with Master: 4bf51bda973949fcb3568cb66538fcdd (127.0.98.1:43063)
I20260619 14:50:10.477299 16324 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:49329
I20260619 14:50:10.480456 16518 heartbeater.cc:503] Master 127.0.98.62:39877 was elected leader, sending a full tablet report...
Jun 19 14:50:10 dist-test-slave-2sb1 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880610, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260619 14:50:10.490451 16525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:10.476321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43063 (local address 127.0.0.1:50618)
0619 14:50:10.476845 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:10.476908 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:10.477083 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:10.478030 (+ 947us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:10.478036 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:10.478513 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:10.478665 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:10.478674 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:10.480305 (+ 1631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:10.480309 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:10.481285 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:10.481295 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:10.481923 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:10.481943 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:10.482081 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:10.482738 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:10.482760 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:10.484485 (+ 1725us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:50:10.488225 (+ 3740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:10.488245 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:10.488251 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:10.488576 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:10.488982 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:10.488986 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:10.488988 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:10.489084 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:10.489641 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:50:10.489648 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:50:10.489800 (+ 152us) client_negotiation.cc:769] Sending connection context
0619 14:50:10.489960 (+ 160us) client_negotiation.cc:240] Negotiation successful
0619 14:50:10.490238 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":165,"threads_started":1}
I20260619 14:50:10.490463 16526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:10.476404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50618 (local address 127.0.98.1:43063)
0619 14:50:10.477543 (+ 1139us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:10.477558 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:10.477586 (+ 28us) server_negotiation.cc:407] Connection header received
0619 14:50:10.477672 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:10.477676 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:10.477858 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:10.478024 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:10.478863 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:10.480139 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:10.481420 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:10.481797 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:10.481939 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:10.484728 (+ 2789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:10.484739 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:10.484750 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:10.484778 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:10.488062 (+ 3284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:10.488762 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:10.488769 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:10.488770 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:10.488867 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:10.489202 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:10.489206 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:10.489208 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:10.489506 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:10.489670 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:10.490072 (+ 402us) server_negotiation.cc:299] Negotiation successful
0619 14:50:10.490291 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":79,"server-negotiator.queue_time_us":902,"thread_start_us":121,"threads_started":1}
I20260619 14:50:10.491745 16452 tablet_service.cc:1514] Processing DeleteTablet for tablet efe8a198d2ea4c39bdd1a51b15bf4c21 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:49:49 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50618
I20260619 14:50:10.492837 16527 tablet_replica.cc:330] stopping tablet replica
I20260619 14:50:10.493227 16527 raft_consensus.cc:2229] T efe8a198d2ea4c39bdd1a51b15bf4c21 P 4bf51bda973949fcb3568cb66538fcdd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:50:10.493536 16527 raft_consensus.cc:2258] T efe8a198d2ea4c39bdd1a51b15bf4c21 P 4bf51bda973949fcb3568cb66538fcdd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:50:10.494903 16527 ts_tablet_manager.cc:1905] T efe8a198d2ea4c39bdd1a51b15bf4c21 P 4bf51bda973949fcb3568cb66538fcdd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:50:10.500820 16527 ts_tablet_manager.cc:1918] T efe8a198d2ea4c39bdd1a51b15bf4c21 P 4bf51bda973949fcb3568cb66538fcdd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:50:10.500955 16527 log.cc:1196] T efe8a198d2ea4c39bdd1a51b15bf4c21 P 4bf51bda973949fcb3568cb66538fcdd: Deleting WAL directory at /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-0/wal/wals/efe8a198d2ea4c39bdd1a51b15bf4c21
I20260619 14:50:10.501247 16527 ts_tablet_manager.cc:1939] T efe8a198d2ea4c39bdd1a51b15bf4c21 P 4bf51bda973949fcb3568cb66538fcdd: Deleting consensus metadata
I20260619 14:50:10.504088 16314 catalog_manager.cc:4841] TS 4bf51bda973949fcb3568cb66538fcdd (127.0.98.1:43063): tablet efe8a198d2ea4c39bdd1a51b15bf4c21 (table test-table [id=b8aa58f2a42842398bcd4020d7a5ae64]) successfully deleted
W20260619 14:50:10.588199 16522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:50:10.589232 16522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:50:10.589341 16522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:50:10.594532 16522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:50:10.594628 16522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:50:10.594695 16522 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 14:50:10.601563 16522 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:46373
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=35715
--tserver_master_addrs=127.0.98.62:39877
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16522
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:50:10.603214 16522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:50:10.604437 16522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:50:10.616408 16532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:10.616783 16533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:10.616783 16535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:50:10.617296 16522 server_base.cc:1034] running on GCE node
I20260619 14:50:10.617820 16522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:50:10.619136 16522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:50:10.622575 16522 hybrid_clock.cc:648] HybridClock initialized: now 1781880610622523 us; error 96 us; skew 500 ppm
Jun 19 14:50:10 dist-test-slave-2sb1 krb5kdc[15746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880610, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:50:10.626430 16522 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 14:50:10.627429 16522 webserver.cc:460] Webserver started at http://127.0.98.2:35715/ using document root <none> and password file <none>
I20260619 14:50:10.628000 16522 fs_manager.cc:359] Metadata directory not provided
I20260619 14:50:10.628077 16522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:50:10.632603 16522 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260619 14:50:10.635476 16542 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:10.637171 16522 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:50:10.637454 16522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-1/wal
uuid: "b9dc81733a644dbe85d11ba50f1f4ebc"
format_stamp: "Formatted at 2026-06-19 14:49:49 on dist-test-slave-2sb1"
server_key: "aa672057f3756cee7397d5b34ba3608d"
server_key_iv: "c0db8c6ce83864fa67b387c353a42fbc"
server_key_version: "encryptionkey@0"
I20260619 14:50:10.638377 16522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:50:10.652967 16522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:50:10.656945 16522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:50:10.657168 16522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:50:10.657871 16522 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:50:10.658829 16522 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:50:10.658878 16522 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:10.658951 16522 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:50:10.658993 16522 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:10.669124 16522 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46373
I20260619 14:50:10.669139 16655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46373 every 8 connection(s)
I20260619 14:50:10.670223 16522 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-1/data/info.pb
I20260619 14:50:10.671128 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 16522
I20260619 14:50:10.671583 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:39361
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:39877
--webserver_port=44599
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 19 14:50:10 dist-test-slave-2sb1 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880610, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:50:10.691632 16520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:10.672824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50225 (local address 127.0.98.62:39877)
0619 14:50:10.673098 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:10.673105 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:10.673859 (+ 754us) server_negotiation.cc:407] Connection header received
0619 14:50:10.676915 (+ 3056us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:10.676919 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:10.677003 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:10.677114 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:10.679774 (+ 2660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:10.680321 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:10.682720 (+ 2399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:10.683035 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:10.683164 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:10.686968 (+ 3804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:10.686972 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:10.686975 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:10.687012 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:10.688902 (+ 1890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:10.689641 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:10.689645 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:10.689646 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:10.689736 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:10.690148 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:10.690150 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:10.690151 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:10.690348 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:10.690471 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:10.691247 (+ 776us) server_negotiation.cc:299] Negotiation successful
0619 14:50:10.691485 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145}
I20260619 14:50:10.692927 16659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:10.673098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39877 (local address 127.0.98.2:50225)
0619 14:50:10.673646 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:10.673715 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:10.676249 (+ 2534us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:10.677441 (+ 1192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:10.677457 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:10.678336 (+ 879us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:10.679496 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:10.679529 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:10.680563 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:10.680568 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:10.682495 (+ 1927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:10.682515 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:10.683267 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:10.683274 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:10.683549 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:10.684586 (+ 1037us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:10.684625 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:10.686695 (+ 2070us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:50:10.689048 (+ 2353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:10.689056 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:10.689077 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:10.689510 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:10.689833 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:10.689844 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:10.689846 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:10.690040 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:10.690491 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:50:10.690499 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:50:10.690899 (+ 400us) client_negotiation.cc:769] Sending connection context
0619 14:50:10.691143 (+ 244us) client_negotiation.cc:240] Negotiation successful
0619 14:50:10.691664 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":118,"threads_started":1}
I20260619 14:50:10.694484 16656 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39877
I20260619 14:50:10.694773 16656 heartbeater.cc:464] Registering TS with master...
I20260619 14:50:10.695436 16656 heartbeater.cc:511] Master 127.0.98.62:39877 requested a full tablet report, sending...
I20260619 14:50:10.696707 16324 ts_manager.cc:194] Registered new tserver with Master: b9dc81733a644dbe85d11ba50f1f4ebc (127.0.98.2:46373)
I20260619 14:50:10.697497 16324 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50225
W20260619 14:50:10.802049 16658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:50:10.802517 16658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:50:10.802580 16658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:50:10.807718 16658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:50:10.807827 16658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:50:10.807900 16658 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 14:50:10.814872 16658 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:39361
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=44599
--tserver_master_addrs=127.0.98.62:39877
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16658
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:50:10.816141 16658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:50:10.817389 16658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:50:10.830211 16664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:10.830211 16665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:10.830205 16667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:50:10.830585 16658 server_base.cc:1034] running on GCE node
I20260619 14:50:10.831166 16658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:50:10.832024 16658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:50:10.833657 16658 hybrid_clock.cc:648] HybridClock initialized: now 1781880610833515 us; error 121 us; skew 500 ppm
Jun 19 14:50:10 dist-test-slave-2sb1 krb5kdc[15746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880610, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:50:10.838404 16658 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 14:50:10.839666 16658 webserver.cc:460] Webserver started at http://127.0.98.3:44599/ using document root <none> and password file <none>
I20260619 14:50:10.840404 16658 fs_manager.cc:359] Metadata directory not provided
I20260619 14:50:10.840529 16658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:50:10.845728 16658 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 14:50:10.848534 16674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:10.850241 16658 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:50:10.850453 16658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-2/wal
uuid: "c25b2be704c74c89a191aa1977240d13"
format_stamp: "Formatted at 2026-06-19 14:49:49 on dist-test-slave-2sb1"
server_key: "a78f727aa1c3c1a6f0023647516cf87a"
server_key_iv: "809fe72fe93ce805a089cdda04a1fc8c"
server_key_version: "encryptionkey@0"
I20260619 14:50:10.851130 16658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:50:10.880605 16658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:50:10.884783 16658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:50:10.885023 16658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:50:10.885705 16658 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:50:10.886627 16658 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:50:10.886677 16658 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:10.886718 16658 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:50:10.886732 16658 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:10.897552 16658 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39361
I20260619 14:50:10.897610 16787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39361 every 8 connection(s)
I20260619 14:50:10.898851 16658 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-2/data/info.pb
I20260619 14:50:10.902148 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 16658
Jun 19 14:50:10 dist-test-slave-2sb1 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880610, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:50:10.913537 16792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:10.903161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54010 (local address 127.0.98.62:39877)
0619 14:50:10.904017 (+ 856us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:10.904021 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:10.904033 (+ 12us) server_negotiation.cc:407] Connection header received
0619 14:50:10.904072 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:10.904074 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:10.904112 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:10.904232 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:10.905176 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:10.905730 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:10.906985 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:10.907234 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:10.907371 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:10.909030 (+ 1659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:10.909034 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:10.909036 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:10.909067 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:10.911510 (+ 2443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:10.912075 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:10.912081 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:10.912086 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:10.912189 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:10.912496 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:10.912500 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:10.912502 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:10.912790 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:10.913059 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:10.913171 (+ 112us) server_negotiation.cc:299] Negotiation successful
0619 14:50:10.913307 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":68,"server-negotiator.queue_time_us":782,"spinlock_wait_cycles":26368,"thread_start_us":172,"threads_started":1}
I20260619 14:50:10.917378 16520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:10.901222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41263 (local address 127.0.98.62:39877)
0619 14:50:10.901406 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:10.901413 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:10.903020 (+ 1607us) server_negotiation.cc:407] Connection header received
0619 14:50:10.904153 (+ 1133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:10.904157 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:10.904206 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:10.904272 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:10.905682 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:10.906521 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:10.908305 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:10.908895 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:10.909050 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:10.912319 (+ 3269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:10.912322 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:10.912324 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:10.912348 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:10.914024 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:10.914771 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:10.914774 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:10.914776 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:10.914871 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:10.915831 (+ 960us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:10.915834 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:10.915835 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:10.916044 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:10.916136 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:10.917064 (+ 928us) server_negotiation.cc:299] Negotiation successful
0619 14:50:10.917207 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":77}
I20260619 14:50:10.918308 16790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:10.901643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39877 (local address 127.0.98.3:41263)
0619 14:50:10.902696 (+ 1053us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:10.902771 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:10.903930 (+ 1159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:10.904492 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:10.904499 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:10.904991 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:10.905524 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:10.905539 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:10.907087 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:10.907091 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:10.908143 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:10.908156 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:10.909106 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:10.909111 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:10.909597 (+ 486us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:10.910302 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:10.910321 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:10.912103 (+ 1782us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:50:10.914192 (+ 2089us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:10.914200 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:10.914221 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:10.914644 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:10.915206 (+ 562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:10.915212 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:10.915214 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:10.915672 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:10.916228 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:50:10.916235 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:50:10.916736 (+ 501us) client_negotiation.cc:769] Sending connection context
0619 14:50:10.916981 (+ 245us) client_negotiation.cc:240] Negotiation successful
0619 14:50:10.917307 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":774,"thread_start_us":181,"threads_started":1}
I20260619 14:50:10.919991 16788 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39877
I20260619 14:50:10.920301 16788 heartbeater.cc:464] Registering TS with master...
I20260619 14:50:10.921077 16788 heartbeater.cc:511] Master 127.0.98.62:39877 requested a full tablet report, sending...
I20260619 14:50:10.922294 16324 ts_manager.cc:194] Registered new tserver with Master: c25b2be704c74c89a191aa1977240d13 (127.0.98.3:39361)
I20260619 14:50:10.923022 16324 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41263
I20260619 14:50:10.928252 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:50:10.933564 16520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:10.929457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54012 (local address 127.0.98.62:39877)
0619 14:50:10.929639 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:10.929644 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:10.929801 (+ 157us) server_negotiation.cc:407] Connection header received
0619 14:50:10.930031 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:10.930033 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:10.930143 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:10.930266 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:50:10.930616 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:10.931305 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:10.932204 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:10.932429 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:10.932546 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:10.932939 (+ 393us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:50:10.933093 (+ 154us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0619 14:50:10.933398 (+ 305us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54012: token expired
Metrics: {"server-negotiator.queue_time_us":78}
W20260619 14:50:10.933667 16520 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54012: token expired
W20260619 14:50:10.934154 16256 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:39877)
I20260619 14:50:10.943836 16520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:10.934644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54026 (local address 127.0.98.62:39877)
0619 14:50:10.934838 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:10.934841 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:10.934856 (+ 15us) server_negotiation.cc:407] Connection header received
0619 14:50:10.934915 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:10.934917 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:10.934955 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:10.935027 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:10.935741 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:10.936267 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:10.937367 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:10.937578 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:10.937661 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:10.939627 (+ 1966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:10.939632 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:10.939636 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:10.939671 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:10.941477 (+ 1806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:10.942063 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:10.942067 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:10.942072 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:10.942196 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:10.942597 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:10.942601 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:10.942603 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:10.942909 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:10.943094 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:10.943479 (+ 385us) server_negotiation.cc:299] Negotiation successful
0619 14:50:10.943630 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260619 14:50:10.946724 16324 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54026:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:50:10.950299 16324 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:50:10.960309 16452 tablet_service.cc:1467] Processing CreateTablet for tablet 4ecc7db119b642069c5f537791704632 (DEFAULT_TABLE table=test-table [id=a6f05cb5d123454aa5b5d8172d85dfe8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:50:10.961099 16452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ecc7db119b642069c5f537791704632. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:50:10.965340 16410 tablet_bootstrap.cc:492] T 4ecc7db119b642069c5f537791704632 P 4bf51bda973949fcb3568cb66538fcdd: Bootstrap starting.
I20260619 14:50:10.966174 16525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:10.958859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46373 (local address 127.0.0.1:50924)
0619 14:50:10.959008 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:10.959026 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:10.959119 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:10.960085 (+ 966us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:10.960088 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:10.960107 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:50:10.960222 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:10.960227 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:10.962553 (+ 2326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:10.962557 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:10.963842 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:10.963853 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:10.965439 (+ 1586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:10.965444 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:10.965562 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:10.965639 (+ 77us) client_negotiation.cc:769] Sending connection context
0619 14:50:10.965722 (+ 83us) client_negotiation.cc:240] Negotiation successful
0619 14:50:10.965996 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56,"mutex_wait_us":189}
I20260619 14:50:10.966431 16798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:10.959142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39361 (local address 127.0.0.1:48500)
0619 14:50:10.959854 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:10.959869 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:10.959985 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:10.960681 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:10.960685 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:10.960707 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:50:10.960808 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:10.960815 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:10.962511 (+ 1696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:10.962515 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:10.964433 (+ 1918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:10.964446 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:10.965722 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:10.965726 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:10.966002 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:10.966074 (+ 72us) client_negotiation.cc:769] Sending connection context
0619 14:50:10.966176 (+ 102us) client_negotiation.cc:240] Negotiation successful
0619 14:50:10.966273 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":646,"mutex_wait_us":56,"thread_start_us":162,"threads_started":1}
I20260619 14:50:10.966540 16799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:10.959433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48500 (local address 127.0.98.3:39361)
0619 14:50:10.959774 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:10.959786 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:10.959986 (+ 200us) server_negotiation.cc:407] Connection header received
0619 14:50:10.960089 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:10.960094 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:10.960291 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:10.960448 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:50:10.960944 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:10.962338 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:10.964633 (+ 2295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:10.965575 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:10.965717 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:10.965878 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:10.966261 (+ 383us) server_negotiation.cc:299] Negotiation successful
0619 14:50:10.966394 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":151,"threads_started":1}
I20260619 14:50:10.966670 16410 tablet_bootstrap.cc:654] T 4ecc7db119b642069c5f537791704632 P 4bf51bda973949fcb3568cb66538fcdd: Neither blocks nor log segments found. Creating new log.
I20260619 14:50:10.967442 16797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:10.959156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50924 (local address 127.0.98.2:46373)
0619 14:50:10.959508 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:10.959522 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:10.959547 (+ 25us) server_negotiation.cc:407] Connection header received
0619 14:50:10.959630 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:10.959636 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:10.959877 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:10.960401 (+ 524us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:50:10.960915 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:10.962403 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:10.964180 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:10.965262 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:10.966901 (+ 1639us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:10.967095 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:10.967160 (+ 65us) server_negotiation.cc:299] Negotiation successful
0619 14:50:10.967290 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":143,"threads_started":1}
I20260619 14:50:10.969905 16410 tablet_bootstrap.cc:492] T 4ecc7db119b642069c5f537791704632 P 4bf51bda973949fcb3568cb66538fcdd: No bootstrap required, opened a new log
I20260619 14:50:10.970031 16410 ts_tablet_manager.cc:1397] T 4ecc7db119b642069c5f537791704632 P 4bf51bda973949fcb3568cb66538fcdd: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260619 14:50:10.970566 16410 raft_consensus.cc:348] T 4ecc7db119b642069c5f537791704632 P 4bf51bda973949fcb3568cb66538fcdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9dc81733a644dbe85d11ba50f1f4ebc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46373 } } peers { permanent_uuid: "c25b2be704c74c89a191aa1977240d13" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39361 } } peers { permanent_uuid: "4bf51bda973949fcb3568cb66538fcdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43063 } }
I20260619 14:50:10.970721 16410 raft_consensus.cc:374] T 4ecc7db119b642069c5f537791704632 P 4bf51bda973949fcb3568cb66538fcdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:50:10.970767 16410 raft_consensus.cc:729] T 4ecc7db119b642069c5f537791704632 P 4bf51bda973949fcb3568cb66538fcdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bf51bda973949fcb3568cb66538fcdd, State: Initialized, Role: FOLLOWER
I20260619 14:50:10.970968 16410 consensus_queue.cc:260] T 4ecc7db119b642069c5f537791704632 P 4bf51bda973949fcb3568cb66538fcdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9dc81733a644dbe85d11ba50f1f4ebc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46373 } } peers { permanent_uuid: "c25b2be704c74c89a191aa1977240d13" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39361 } } peers { permanent_uuid: "4bf51bda973949fcb3568cb66538fcdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43063 } }
I20260619 14:50:10.970633 16722 tablet_service.cc:1467] Processing CreateTablet for tablet 4ecc7db119b642069c5f537791704632 (DEFAULT_TABLE table=test-table [id=a6f05cb5d123454aa5b5d8172d85dfe8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:50:10.970570 16590 tablet_service.cc:1467] Processing CreateTablet for tablet 4ecc7db119b642069c5f537791704632 (DEFAULT_TABLE table=test-table [id=a6f05cb5d123454aa5b5d8172d85dfe8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:50:10.971380 16410 ts_tablet_manager.cc:1428] T 4ecc7db119b642069c5f537791704632 P 4bf51bda973949fcb3568cb66538fcdd: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260619 14:50:10.971859 16590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ecc7db119b642069c5f537791704632. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:50:10.971892 16722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ecc7db119b642069c5f537791704632. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:50:10.980298 16802 tablet_bootstrap.cc:492] T 4ecc7db119b642069c5f537791704632 P c25b2be704c74c89a191aa1977240d13: Bootstrap starting.
I20260619 14:50:10.981030 16803 tablet_bootstrap.cc:492] T 4ecc7db119b642069c5f537791704632 P b9dc81733a644dbe85d11ba50f1f4ebc: Bootstrap starting.
I20260619 14:50:10.983045 16802 tablet_bootstrap.cc:654] T 4ecc7db119b642069c5f537791704632 P c25b2be704c74c89a191aa1977240d13: Neither blocks nor log segments found. Creating new log.
I20260619 14:50:10.983449 16803 tablet_bootstrap.cc:654] T 4ecc7db119b642069c5f537791704632 P b9dc81733a644dbe85d11ba50f1f4ebc: Neither blocks nor log segments found. Creating new log.
I20260619 14:50:10.984223 16802 log.cc:824] T 4ecc7db119b642069c5f537791704632 P c25b2be704c74c89a191aa1977240d13: Log is configured to *not* fsync() on all Append() calls
I20260619 14:50:10.984225 16803 log.cc:824] T 4ecc7db119b642069c5f537791704632 P b9dc81733a644dbe85d11ba50f1f4ebc: Log is configured to *not* fsync() on all Append() calls
I20260619 14:50:10.986634 16802 tablet_bootstrap.cc:492] T 4ecc7db119b642069c5f537791704632 P c25b2be704c74c89a191aa1977240d13: No bootstrap required, opened a new log
I20260619 14:50:10.986697 16803 tablet_bootstrap.cc:492] T 4ecc7db119b642069c5f537791704632 P b9dc81733a644dbe85d11ba50f1f4ebc: No bootstrap required, opened a new log
I20260619 14:50:10.986950 16802 ts_tablet_manager.cc:1397] T 4ecc7db119b642069c5f537791704632 P c25b2be704c74c89a191aa1977240d13: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260619 14:50:10.986963 16803 ts_tablet_manager.cc:1397] T 4ecc7db119b642069c5f537791704632 P b9dc81733a644dbe85d11ba50f1f4ebc: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260619 14:50:10.990548 16802 raft_consensus.cc:348] T 4ecc7db119b642069c5f537791704632 P c25b2be704c74c89a191aa1977240d13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9dc81733a644dbe85d11ba50f1f4ebc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46373 } } peers { permanent_uuid: "c25b2be704c74c89a191aa1977240d13" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39361 } } peers { permanent_uuid: "4bf51bda973949fcb3568cb66538fcdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43063 } }
I20260619 14:50:10.991010 16802 raft_consensus.cc:374] T 4ecc7db119b642069c5f537791704632 P c25b2be704c74c89a191aa1977240d13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:50:10.991091 16802 raft_consensus.cc:729] T 4ecc7db119b642069c5f537791704632 P c25b2be704c74c89a191aa1977240d13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c25b2be704c74c89a191aa1977240d13, State: Initialized, Role: FOLLOWER
I20260619 14:50:10.990707 16803 raft_consensus.cc:348] T 4ecc7db119b642069c5f537791704632 P b9dc81733a644dbe85d11ba50f1f4ebc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9dc81733a644dbe85d11ba50f1f4ebc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46373 } } peers { permanent_uuid: "c25b2be704c74c89a191aa1977240d13" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39361 } } peers { permanent_uuid: "4bf51bda973949fcb3568cb66538fcdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43063 } }
I20260619 14:50:10.991255 16803 raft_consensus.cc:374] T 4ecc7db119b642069c5f537791704632 P b9dc81733a644dbe85d11ba50f1f4ebc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:50:10.991343 16803 raft_consensus.cc:729] T 4ecc7db119b642069c5f537791704632 P b9dc81733a644dbe85d11ba50f1f4ebc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9dc81733a644dbe85d11ba50f1f4ebc, State: Initialized, Role: FOLLOWER
I20260619 14:50:10.991699 16802 consensus_queue.cc:260] T 4ecc7db119b642069c5f537791704632 P c25b2be704c74c89a191aa1977240d13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9dc81733a644dbe85d11ba50f1f4ebc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46373 } } peers { permanent_uuid: "c25b2be704c74c89a191aa1977240d13" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39361 } } peers { permanent_uuid: "4bf51bda973949fcb3568cb66538fcdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43063 } }
I20260619 14:50:10.991756 16803 consensus_queue.cc:260] T 4ecc7db119b642069c5f537791704632 P b9dc81733a644dbe85d11ba50f1f4ebc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9dc81733a644dbe85d11ba50f1f4ebc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46373 } } peers { permanent_uuid: "c25b2be704c74c89a191aa1977240d13" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39361 } } peers { permanent_uuid: "4bf51bda973949fcb3568cb66538fcdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43063 } }
I20260619 14:50:10.992470 16788 heartbeater.cc:503] Master 127.0.98.62:39877 was elected leader, sending a full tablet report...
I20260619 14:50:10.992501 16656 heartbeater.cc:503] Master 127.0.98.62:39877 was elected leader, sending a full tablet report...
I20260619 14:50:10.992894 16802 ts_tablet_manager.cc:1428] T 4ecc7db119b642069c5f537791704632 P c25b2be704c74c89a191aa1977240d13: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260619 14:50:10.992892 16803 ts_tablet_manager.cc:1428] T 4ecc7db119b642069c5f537791704632 P b9dc81733a644dbe85d11ba50f1f4ebc: Time spent starting tablet: real 0.006s user 0.004s sys 0.003s
I20260619 14:50:11.109211 16806 raft_consensus.cc:482] T 4ecc7db119b642069c5f537791704632 P c25b2be704c74c89a191aa1977240d13 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:50:11.109447 16806 raft_consensus.cc:504] T 4ecc7db119b642069c5f537791704632 P c25b2be704c74c89a191aa1977240d13 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9dc81733a644dbe85d11ba50f1f4ebc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46373 } } peers { permanent_uuid: "c25b2be704c74c89a191aa1977240d13" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39361 } } peers { permanent_uuid: "4bf51bda973949fcb3568cb66538fcdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43063 } }
I20260619 14:50:11.110702 16806 leader_election.cc:288] T 4ecc7db119b642069c5f537791704632 P c25b2be704c74c89a191aa1977240d13 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b9dc81733a644dbe85d11ba50f1f4ebc (127.0.98.2:46373), 4bf51bda973949fcb3568cb66538fcdd (127.0.98.1:43063)
I20260619 14:50:11.116743 16797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:11.111166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57567 (local address 127.0.98.2:46373)
0619 14:50:11.111334 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:11.111348 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:11.112194 (+ 846us) server_negotiation.cc:407] Connection header received
0619 14:50:11.112394 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:11.112398 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:11.112472 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:11.112564 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:50:11.112988 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:11.113713 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:11.115358 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:11.115912 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:11.116018 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:11.116080 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:11.116383 (+ 303us) server_negotiation.cc:299] Negotiation successful
0619 14:50:11.116533 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 14:50:11.116760 16809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:11.111051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46373 (local address 127.0.98.3:57567)
0619 14:50:11.112105 (+ 1054us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:11.112120 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:11.112247 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:11.112680 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:11.112684 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:11.112704 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:50:11.112836 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:11.112843 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:11.113859 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:11.113863 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:11.115195 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:11.115208 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:11.116076 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:11.116081 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:11.116189 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:11.116225 (+ 36us) client_negotiation.cc:769] Sending connection context
0619 14:50:11.116312 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 14:50:11.116505 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":960,"mutex_wait_us":5,"thread_start_us":675,"threads_started":1}
I20260619 14:50:11.117012 16808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:11.111237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43119 (local address 127.0.98.1:43063)
0619 14:50:11.111588 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:11.111591 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:11.111606 (+ 15us) server_negotiation.cc:407] Connection header received
0619 14:50:11.111900 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:11.111903 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:11.111976 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:11.112445 (+ 469us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:50:11.112581 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:11.113437 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:11.115406 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:11.116170 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:11.116315 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:11.116702 (+ 387us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:11.116759 (+ 57us) server_negotiation.cc:299] Negotiation successful
0619 14:50:11.116861 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":144,"threads_started":1}
I20260619 14:50:11.117283 16790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:11.111009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43063 (local address 127.0.98.3:43119)
0619 14:50:11.111142 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:11.111161 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:11.111714 (+ 553us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:11.112131 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:11.112137 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:11.112161 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:50:11.112309 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:11.112318 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:11.113631 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:11.113636 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:11.115241 (+ 1605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:11.115255 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:11.116389 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:11.116394 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:11.116518 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:11.116541 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 14:50:11.116629 (+ 88us) client_negotiation.cc:240] Negotiation successful
0619 14:50:11.116730 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":98}
I20260619 14:50:11.118410 16610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ecc7db119b642069c5f537791704632" candidate_uuid: "c25b2be704c74c89a191aa1977240d13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9dc81733a644dbe85d11ba50f1f4ebc" is_pre_election: true
I20260619 14:50:11.118448 16472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ecc7db119b642069c5f537791704632" candidate_uuid: "c25b2be704c74c89a191aa1977240d13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bf51bda973949fcb3568cb66538fcdd" is_pre_election: true
I20260619 14:50:11.118953 16472 raft_consensus.cc:2454] T 4ecc7db119b642069c5f537791704632 P 4bf51bda973949fcb3568cb66538fcdd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c25b2be704c74c89a191aa1977240d13 in term 0.
I20260619 14:50:11.118973 16610 raft_consensus.cc:2454] T 4ecc7db119b642069c5f537791704632 P b9dc81733a644dbe85d11ba50f1f4ebc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c25b2be704c74c89a191aa1977240d13 in term 0.
I20260619 14:50:11.119848 16678 leader_election.cc:302] T 4ecc7db119b642069c5f537791704632 P c25b2be704c74c89a191aa1977240d13 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4bf51bda973949fcb3568cb66538fcdd, b9dc81733a644dbe85d11ba50f1f4ebc, c25b2be704c74c89a191aa1977240d13; no voters:
I20260619 14:50:11.120911 16806 raft_consensus.cc:2781] T 4ecc7db119b642069c5f537791704632 P c25b2be704c74c89a191aa1977240d13 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:50:11.121215 16806 raft_consensus.cc:482] T 4ecc7db119b642069c5f537791704632 P c25b2be704c74c89a191aa1977240d13 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:50:11.121282 16806 raft_consensus.cc:3037] T 4ecc7db119b642069c5f537791704632 P c25b2be704c74c89a191aa1977240d13 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:50:11.122681 16806 raft_consensus.cc:504] T 4ecc7db119b642069c5f537791704632 P c25b2be704c74c89a191aa1977240d13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9dc81733a644dbe85d11ba50f1f4ebc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46373 } } peers { permanent_uuid: "c25b2be704c74c89a191aa1977240d13" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39361 } } peers { permanent_uuid: "4bf51bda973949fcb3568cb66538fcdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43063 } }
I20260619 14:50:11.123217 16806 leader_election.cc:288] T 4ecc7db119b642069c5f537791704632 P c25b2be704c74c89a191aa1977240d13 [CANDIDATE]: Term 1 election: Requested vote from peers b9dc81733a644dbe85d11ba50f1f4ebc (127.0.98.2:46373), 4bf51bda973949fcb3568cb66538fcdd (127.0.98.1:43063)
I20260619 14:50:11.123857 16472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ecc7db119b642069c5f537791704632" candidate_uuid: "c25b2be704c74c89a191aa1977240d13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bf51bda973949fcb3568cb66538fcdd"
I20260619 14:50:11.123849 16610 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ecc7db119b642069c5f537791704632" candidate_uuid: "c25b2be704c74c89a191aa1977240d13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9dc81733a644dbe85d11ba50f1f4ebc"
I20260619 14:50:11.124204 16472 raft_consensus.cc:3037] T 4ecc7db119b642069c5f537791704632 P 4bf51bda973949fcb3568cb66538fcdd [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:50:11.124221 16610 raft_consensus.cc:3037] T 4ecc7db119b642069c5f537791704632 P b9dc81733a644dbe85d11ba50f1f4ebc [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:50:11.125893 16610 raft_consensus.cc:2454] T 4ecc7db119b642069c5f537791704632 P b9dc81733a644dbe85d11ba50f1f4ebc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c25b2be704c74c89a191aa1977240d13 in term 1.
I20260619 14:50:11.125893 16472 raft_consensus.cc:2454] T 4ecc7db119b642069c5f537791704632 P 4bf51bda973949fcb3568cb66538fcdd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c25b2be704c74c89a191aa1977240d13 in term 1.
I20260619 14:50:11.126657 16675 leader_election.cc:302] T 4ecc7db119b642069c5f537791704632 P c25b2be704c74c89a191aa1977240d13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bf51bda973949fcb3568cb66538fcdd, c25b2be704c74c89a191aa1977240d13; no voters:
I20260619 14:50:11.126904 16806 raft_consensus.cc:2781] T 4ecc7db119b642069c5f537791704632 P c25b2be704c74c89a191aa1977240d13 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:50:11.127161 16806 raft_consensus.cc:686] T 4ecc7db119b642069c5f537791704632 P c25b2be704c74c89a191aa1977240d13 [term 1 LEADER]: Becoming Leader. State: Replica: c25b2be704c74c89a191aa1977240d13, State: Running, Role: LEADER
I20260619 14:50:11.128022 16806 consensus_queue.cc:237] T 4ecc7db119b642069c5f537791704632 P c25b2be704c74c89a191aa1977240d13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9dc81733a644dbe85d11ba50f1f4ebc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46373 } } peers { permanent_uuid: "c25b2be704c74c89a191aa1977240d13" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39361 } } peers { permanent_uuid: "4bf51bda973949fcb3568cb66538fcdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43063 } }
I20260619 14:50:11.132112 16330 catalog_manager.cc:5495] T 4ecc7db119b642069c5f537791704632 P c25b2be704c74c89a191aa1977240d13 reported cstate change: term changed from 0 to 1, leader changed from <none> to c25b2be704c74c89a191aa1977240d13 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "c25b2be704c74c89a191aa1977240d13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9dc81733a644dbe85d11ba50f1f4ebc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c25b2be704c74c89a191aa1977240d13" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39361 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4bf51bda973949fcb3568cb66538fcdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43063 } health_report { overall_health: UNKNOWN } } }
W20260619 14:50:11.150123 16789 tablet.cc:2369] T 4ecc7db119b642069c5f537791704632 P c25b2be704c74c89a191aa1977240d13: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:50:11.171077 16799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:11.166907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48516 (local address 127.0.98.3:39361)
0619 14:50:11.167141 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:11.167146 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:11.167266 (+ 120us) server_negotiation.cc:407] Connection header received
0619 14:50:11.167581 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:11.167584 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:11.167644 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:11.167899 (+ 255us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:50:11.168019 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:11.168536 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:11.169429 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:11.169726 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:11.169847 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:11.170199 (+ 352us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:50:11.170645 (+ 446us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:50:11.170793 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:11.170829 (+ 36us) server_negotiation.cc:299] Negotiation successful
0619 14:50:11.170893 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
W20260619 14:50:11.171517 16657 tablet.cc:2369] T 4ecc7db119b642069c5f537791704632 P b9dc81733a644dbe85d11ba50f1f4ebc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:50:11.178929 16610 raft_consensus.cc:1261] T 4ecc7db119b642069c5f537791704632 P b9dc81733a644dbe85d11ba50f1f4ebc [term 1 FOLLOWER]: Refusing update from remote peer c25b2be704c74c89a191aa1977240d13: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:50:11.179015 16472 raft_consensus.cc:1261] T 4ecc7db119b642069c5f537791704632 P 4bf51bda973949fcb3568cb66538fcdd [term 1 FOLLOWER]: Refusing update from remote peer c25b2be704c74c89a191aa1977240d13: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:50:11.179754 16810 consensus_queue.cc:1035] T 4ecc7db119b642069c5f537791704632 P c25b2be704c74c89a191aa1977240d13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9dc81733a644dbe85d11ba50f1f4ebc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:50:11.179975 16806 consensus_queue.cc:1035] T 4ecc7db119b642069c5f537791704632 P c25b2be704c74c89a191aa1977240d13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bf51bda973949fcb3568cb66538fcdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43063 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:50:11.188841 16812 mvcc.cc:205] Tried to move back new op lower bound from 7298582983379640320 to 7298582983183769600. Current Snapshot: MvccSnapshot[applied={T|T < 7298582983379640320}]
I20260619 14:50:11.189962 16814 mvcc.cc:205] Tried to move back new op lower bound from 7298582983379640320 to 7298582983183769600. Current Snapshot: MvccSnapshot[applied={T|T < 7298582983379640320}]
I20260619 14:50:11.194978 16813 mvcc.cc:205] Tried to move back new op lower bound from 7298582983379640320 to 7298582983183769600. Current Snapshot: MvccSnapshot[applied={T|T < 7298582983379640320}]
I20260619 14:50:11.202021 16330 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54026:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:50:11.202266 16330 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54026:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:50:11.205215 16330 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 85f59d8a5c854579995d8ebfc3501da5: Sending DeleteTablet for 3 replicas of tablet 4ecc7db119b642069c5f537791704632
I20260619 14:50:11.206069 16722 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ecc7db119b642069c5f537791704632 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:50:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48500
I20260619 14:50:11.206089 16590 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ecc7db119b642069c5f537791704632 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:50:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50924
I20260619 14:50:11.206156 16452 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ecc7db119b642069c5f537791704632 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:50:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50618
I20260619 14:50:11.206664 16823 tablet_replica.cc:330] T 4ecc7db119b642069c5f537791704632 P c25b2be704c74c89a191aa1977240d13: stopping tablet replica
I20260619 14:50:11.206664 16822 tablet_replica.cc:330] T 4ecc7db119b642069c5f537791704632 P b9dc81733a644dbe85d11ba50f1f4ebc: stopping tablet replica
I20260619 14:50:11.207031 16823 raft_consensus.cc:2229] T 4ecc7db119b642069c5f537791704632 P c25b2be704c74c89a191aa1977240d13 [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:50:11.207031 16822 raft_consensus.cc:2229] T 4ecc7db119b642069c5f537791704632 P b9dc81733a644dbe85d11ba50f1f4ebc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:50:11.207244 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 16368
I20260619 14:50:11.207413 16823 raft_consensus.cc:2258] T 4ecc7db119b642069c5f537791704632 P c25b2be704c74c89a191aa1977240d13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:50:11.208029 16822 raft_consensus.cc:2258] T 4ecc7db119b642069c5f537791704632 P b9dc81733a644dbe85d11ba50f1f4ebc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:50:11.209684 16823 ts_tablet_manager.cc:1905] T 4ecc7db119b642069c5f537791704632 P c25b2be704c74c89a191aa1977240d13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:50:11.210384 16822 ts_tablet_manager.cc:1905] T 4ecc7db119b642069c5f537791704632 P b9dc81733a644dbe85d11ba50f1f4ebc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:50:11.213107 16823 ts_tablet_manager.cc:1918] T 4ecc7db119b642069c5f537791704632 P c25b2be704c74c89a191aa1977240d13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:50:11.213255 16823 log.cc:1196] T 4ecc7db119b642069c5f537791704632 P c25b2be704c74c89a191aa1977240d13: Deleting WAL directory at /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-2/wal/wals/4ecc7db119b642069c5f537791704632
I20260619 14:50:11.213537 16822 ts_tablet_manager.cc:1918] T 4ecc7db119b642069c5f537791704632 P b9dc81733a644dbe85d11ba50f1f4ebc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:50:11.213630 16823 ts_tablet_manager.cc:1939] T 4ecc7db119b642069c5f537791704632 P c25b2be704c74c89a191aa1977240d13: Deleting consensus metadata
I20260619 14:50:11.213779 16822 log.cc:1196] T 4ecc7db119b642069c5f537791704632 P b9dc81733a644dbe85d11ba50f1f4ebc: Deleting WAL directory at /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-1/wal/wals/4ecc7db119b642069c5f537791704632
I20260619 14:50:11.214118 16822 ts_tablet_manager.cc:1939] T 4ecc7db119b642069c5f537791704632 P b9dc81733a644dbe85d11ba50f1f4ebc: Deleting consensus metadata
I20260619 14:50:11.215561 16317 catalog_manager.cc:4841] TS b9dc81733a644dbe85d11ba50f1f4ebc (127.0.98.2:46373): tablet 4ecc7db119b642069c5f537791704632 (table test-table [id=a6f05cb5d123454aa5b5d8172d85dfe8]) successfully deleted
W20260619 14:50:11.216478 16314 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43063 (error 108)
I20260619 14:50:11.216773 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 16522
I20260619 14:50:11.216796 16315 catalog_manager.cc:4841] TS c25b2be704c74c89a191aa1977240d13 (127.0.98.3:39361): tablet 4ecc7db119b642069c5f537791704632 (table test-table [id=a6f05cb5d123454aa5b5d8172d85dfe8]) successfully deleted
I20260619 14:50:11.218086 16798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:11.217694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43063 (local address 127.0.0.1:50624)
0619 14:50:11.217899 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:11.217976 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
W20260619 14:50:11.218405 16314 catalog_manager.cc:4584] TS 4bf51bda973949fcb3568cb66538fcdd (127.0.98.1:43063): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4ecc7db119b642069c5f537791704632: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43063: connect: Connection refused (error 111)
I20260619 14:50:11.223443 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 16658
I20260619 14:50:11.232543 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 16297
I20260619 14:50:31.240566 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39877
--webserver_interface=127.0.98.62
--webserver_port=35415
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39877
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:50:31.364379 16825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:50:31.364814 16825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:50:31.364869 16825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:50:31.370049 16825 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260619 14:50:31.370146 16825 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260619 14:50:31.370162 16825 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260619 14:50:31.370177 16825 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:50:31.370198 16825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:50:31.370223 16825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:50:31.370239 16825 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:50:31.370256 16825 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:50:31.377220 16825 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39877
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=35415
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16825
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:50:31.378444 16825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:50:31.379561 16825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:50:31.390102 16833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:50:31.390184 16825 server_base.cc:1034] running on GCE node
W20260619 14:50:31.390045 16830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:31.390053 16831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:50:31.391041 16825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:50:31.392267 16825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:50:31.393465 16825 hybrid_clock.cc:648] HybridClock initialized: now 1781880631393464 us; error 81 us; skew 500 ppm
Jun 19 14:50:31 dist-test-slave-2sb1 krb5kdc[15746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880631, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:50:31.396976 16825 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 14:50:31.398056 16825 webserver.cc:460] Webserver started at http://127.0.98.62:35415/ using document root <none> and password file <none>
I20260619 14:50:31.398708 16825 fs_manager.cc:359] Metadata directory not provided
I20260619 14:50:31.398780 16825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:50:31.403448 16825 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:50:31.407567 16840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:31.409246 16825 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 14:50:31.409409 16825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/master-0/wal
uuid: "85f59d8a5c854579995d8ebfc3501da5"
format_stamp: "Formatted at 2026-06-19 14:49:48 on dist-test-slave-2sb1"
server_key: "bdf7dd6b4385ff46418d8542894d08c3"
server_key_iv: "0d22b10314e11f96a24cb8d704875a66"
server_key_version: "encryptionkey@0"
I20260619 14:50:31.410032 16825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:50:31.430378 16825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:50:31.434471 16825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:50:31.434741 16825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:50:31.443812 16825 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39877
I20260619 14:50:31.443842 16892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39877 every 8 connection(s)
I20260619 14:50:31.444989 16825 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:50:31.449782 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 16825
I20260619 14:50:31.450156 16893 sys_catalog.cc:263] Verifying existing consensus state
I20260619 14:50:31.450356 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:43063
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:39877
--webserver_port=32955
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:50:31.452090 16893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85f59d8a5c854579995d8ebfc3501da5: Bootstrap starting.
I20260619 14:50:31.470415 16893 log.cc:824] T 00000000000000000000000000000000 P 85f59d8a5c854579995d8ebfc3501da5: Log is configured to *not* fsync() on all Append() calls
I20260619 14:50:31.482882 16893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85f59d8a5c854579995d8ebfc3501da5: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 14:50:31.483361 16893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85f59d8a5c854579995d8ebfc3501da5: Bootstrap complete.
I20260619 14:50:31.487893 16893 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85f59d8a5c854579995d8ebfc3501da5 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85f59d8a5c854579995d8ebfc3501da5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39877 } }
I20260619 14:50:31.488425 16893 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85f59d8a5c854579995d8ebfc3501da5 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f59d8a5c854579995d8ebfc3501da5, State: Initialized, Role: FOLLOWER
I20260619 14:50:31.489105 16893 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85f59d8a5c854579995d8ebfc3501da5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85f59d8a5c854579995d8ebfc3501da5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39877 } }
I20260619 14:50:31.489279 16893 raft_consensus.cc:388] T 00000000000000000000000000000000 P 85f59d8a5c854579995d8ebfc3501da5 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:50:31.489377 16893 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85f59d8a5c854579995d8ebfc3501da5 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:50:31.489482 16893 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85f59d8a5c854579995d8ebfc3501da5 [term 2 FOLLOWER]: Advancing to term 3
I20260619 14:50:31.491621 16893 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85f59d8a5c854579995d8ebfc3501da5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85f59d8a5c854579995d8ebfc3501da5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39877 } }
I20260619 14:50:31.492012 16893 leader_election.cc:302] T 00000000000000000000000000000000 P 85f59d8a5c854579995d8ebfc3501da5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85f59d8a5c854579995d8ebfc3501da5; no voters:
I20260619 14:50:31.492420 16893 leader_election.cc:288] T 00000000000000000000000000000000 P 85f59d8a5c854579995d8ebfc3501da5 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260619 14:50:31.492522 16897 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 85f59d8a5c854579995d8ebfc3501da5 [term 3 FOLLOWER]: Leader election won for term 3
I20260619 14:50:31.492761 16897 raft_consensus.cc:686] T 00000000000000000000000000000000 P 85f59d8a5c854579995d8ebfc3501da5 [term 3 LEADER]: Becoming Leader. State: Replica: 85f59d8a5c854579995d8ebfc3501da5, State: Running, Role: LEADER
I20260619 14:50:31.493146 16897 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85f59d8a5c854579995d8ebfc3501da5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85f59d8a5c854579995d8ebfc3501da5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39877 } }
I20260619 14:50:31.493839 16893 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85f59d8a5c854579995d8ebfc3501da5 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:50:31.494745 16899 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85f59d8a5c854579995d8ebfc3501da5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85f59d8a5c854579995d8ebfc3501da5. Latest consensus state: current_term: 3 leader_uuid: "85f59d8a5c854579995d8ebfc3501da5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85f59d8a5c854579995d8ebfc3501da5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39877 } } }
I20260619 14:50:31.494771 16898 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85f59d8a5c854579995d8ebfc3501da5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "85f59d8a5c854579995d8ebfc3501da5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85f59d8a5c854579995d8ebfc3501da5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39877 } } }
I20260619 14:50:31.494884 16899 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85f59d8a5c854579995d8ebfc3501da5 [sys.catalog]: This master's current role is: LEADER
I20260619 14:50:31.494896 16898 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85f59d8a5c854579995d8ebfc3501da5 [sys.catalog]: This master's current role is: LEADER
I20260619 14:50:31.495285 16905 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:50:31.501933 16905 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:50:31.502946 16905 catalog_manager.cc:1259] Loaded cluster ID: 3837edbdb3314120983e94b9eea509df
I20260619 14:50:31.503036 16905 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:50:31.505271 16905 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:50:31.505968 16905 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 85f59d8a5c854579995d8ebfc3501da5: Loaded TSK: 0
I20260619 14:50:31.506069 16905 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 85f59d8a5c854579995d8ebfc3501da5: Loaded TSK: 1
F20260619 14:50:31.506724 16905 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f83671779ad google::LogMessage::Fail() at ??:0
@ 0x7f836717a88a google::LogMessage::SendToLog() at ??:0
@ 0x7f83671774de google::LogMessage::Flush() at ??:0
@ 0x7f8367178a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f8369e3b18d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f8369e33f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f8371adc7bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f8371ab699c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f8371ab7178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f8371af8df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f8371b1782b std::function<>::operator()() at ??:0
@ 0x7f8371ab6b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f8371ab7f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f8371ab4579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f8371af8640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f83693c93d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f83693c9c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f83693cb3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f83693ba512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f836a9fe6db start_thread at ??:0
@ 0x7f8365c7561f clone at ??:0
W20260619 14:50:31.619971 16895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:50:31.620321 16895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:50:31.620429 16895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:50:31.625787 16895 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:50:31.625896 16895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:50:31.625993 16895 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 14:50:31.632382 16895 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:43063
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=32955
--tserver_master_addrs=127.0.98.62:39877
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16895
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:50:31.633683 16895 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:50:31.634756 16895 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:50:31.644917 16940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:31.644920 16943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:50:31.645393 16895 server_base.cc:1034] running on GCE node
W20260619 14:50:31.645423 16941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:50:31.645993 16895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:50:31.646755 16895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:50:31.648101 16895 hybrid_clock.cc:648] HybridClock initialized: now 1781880631648017 us; error 40 us; skew 500 ppm
Jun 19 14:50:31 dist-test-slave-2sb1 krb5kdc[15746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880631, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:50:31.651655 16895 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 14:50:31.653015 16895 webserver.cc:460] Webserver started at http://127.0.98.1:32955/ using document root <none> and password file <none>
I20260619 14:50:31.653676 16895 fs_manager.cc:359] Metadata directory not provided
I20260619 14:50:31.653750 16895 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:50:31.658654 16895 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260619 14:50:31.661383 16950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:31.663090 16895 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:50:31.663304 16895 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-0/wal
uuid: "4bf51bda973949fcb3568cb66538fcdd"
format_stamp: "Formatted at 2026-06-19 14:49:49 on dist-test-slave-2sb1"
server_key: "abb9f876c53f30d1f950e3f76a32d324"
server_key_iv: "f4a116fee231edd0276c783399b9b061"
server_key_version: "encryptionkey@0"
I20260619 14:50:31.664132 16895 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:50:31.684829 16895 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:50:31.688419 16895 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:50:31.688691 16895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:50:31.689399 16895 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:50:31.690613 16957 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260619 14:50:31.694877 16895 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260619 14:50:31.694960 16895 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260619 14:50:31.695013 16895 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260619 14:50:31.696869 16895 ts_tablet_manager.cc:610] Registered 1 tablets
I20260619 14:50:31.696941 16895 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260619 14:50:31.697207 16957 tablet_bootstrap.cc:492] T 4ecc7db119b642069c5f537791704632 P 4bf51bda973949fcb3568cb66538fcdd: Bootstrap starting.
I20260619 14:50:31.708405 16895 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43063
I20260619 14:50:31.708868 17064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43063 every 8 connection(s)
I20260619 14:50:31.710013 16895 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-0/data/info.pb
I20260619 14:50:31.719182 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 16895
I20260619 14:50:31.719600 16957 log.cc:824] T 4ecc7db119b642069c5f537791704632 P 4bf51bda973949fcb3568cb66538fcdd: Log is configured to *not* fsync() on all Append() calls
I20260619 14:50:31.719763 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:46373
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:39877
--webserver_port=35715
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:50:31.728760 16957 tablet_bootstrap.cc:492] T 4ecc7db119b642069c5f537791704632 P 4bf51bda973949fcb3568cb66538fcdd: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260619 14:50:31.729238 16957 tablet_bootstrap.cc:492] T 4ecc7db119b642069c5f537791704632 P 4bf51bda973949fcb3568cb66538fcdd: Bootstrap complete.
I20260619 14:50:31.730665 16957 ts_tablet_manager.cc:1397] T 4ecc7db119b642069c5f537791704632 P 4bf51bda973949fcb3568cb66538fcdd: Time spent bootstrapping tablet: real 0.034s user 0.026s sys 0.005s
I20260619 14:50:31.733659 16957 raft_consensus.cc:348] T 4ecc7db119b642069c5f537791704632 P 4bf51bda973949fcb3568cb66538fcdd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9dc81733a644dbe85d11ba50f1f4ebc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46373 } } peers { permanent_uuid: "c25b2be704c74c89a191aa1977240d13" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39361 } } peers { permanent_uuid: "4bf51bda973949fcb3568cb66538fcdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43063 } }
I20260619 14:50:31.734175 16957 raft_consensus.cc:729] T 4ecc7db119b642069c5f537791704632 P 4bf51bda973949fcb3568cb66538fcdd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bf51bda973949fcb3568cb66538fcdd, State: Initialized, Role: FOLLOWER
I20260619 14:50:31.734732 16957 consensus_queue.cc:260] T 4ecc7db119b642069c5f537791704632 P 4bf51bda973949fcb3568cb66538fcdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9dc81733a644dbe85d11ba50f1f4ebc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46373 } } peers { permanent_uuid: "c25b2be704c74c89a191aa1977240d13" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39361 } } peers { permanent_uuid: "4bf51bda973949fcb3568cb66538fcdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43063 } }
I20260619 14:50:31.735823 16957 ts_tablet_manager.cc:1428] T 4ecc7db119b642069c5f537791704632 P 4bf51bda973949fcb3568cb66538fcdd: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260619 14:50:31.780952 17067 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:50:31.713216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39877 (local address 127.0.98.1:33235)
0619 14:50:31.713937 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:31.714018 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:31.715207 (+ 1189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:31.779898 (+ 64691us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39877: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":204,"threads_started":1}
W20260619 14:50:31.781635 17065 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39877 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:39877: Client connection negotiation failed: client connection to 127.0.98.62:39877: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260619 14:50:31.782509 17067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:31.782185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39877 (local address 127.0.98.1:51045)
0619 14:50:31.782343 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:31.782426 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260619 14:50:31.783210 17067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:31.782972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39877 (local address 127.0.98.1:44349)
0619 14:50:31.783092 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:31.783145 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260619 14:50:31.783397 17065 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 14:50:31.868332 17069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:50:31.868883 17069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:50:31.868979 17069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:50:31.874189 17069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:50:31.874298 17069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:50:31.874415 17069 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 14:50:31.881526 17069 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:46373
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=35715
--tserver_master_addrs=127.0.98.62:39877
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17069
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:50:31.882882 17069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:50:31.884027 17069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 14:50:31.895274 17069 server_base.cc:1034] running on GCE node
W20260619 14:50:31.895255 17076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:31.895371 17078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:31.895301 17075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:50:31.896170 17069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:50:31.897086 17069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:50:31.898329 17069 hybrid_clock.cc:648] HybridClock initialized: now 1781880631898309 us; error 33 us; skew 500 ppm
Jun 19 14:50:31 dist-test-slave-2sb1 krb5kdc[15746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880631, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:50:31.902040 17069 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 14:50:31.903167 17069 webserver.cc:460] Webserver started at http://127.0.98.2:35715/ using document root <none> and password file <none>
I20260619 14:50:31.903815 17069 fs_manager.cc:359] Metadata directory not provided
I20260619 14:50:31.903893 17069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:50:31.908773 17069 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260619 14:50:31.911443 17085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:31.913085 17069 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:50:31.913231 17069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-1/wal
uuid: "b9dc81733a644dbe85d11ba50f1f4ebc"
format_stamp: "Formatted at 2026-06-19 14:49:49 on dist-test-slave-2sb1"
server_key: "aa672057f3756cee7397d5b34ba3608d"
server_key_iv: "c0db8c6ce83864fa67b387c353a42fbc"
server_key_version: "encryptionkey@0"
I20260619 14:50:31.913883 17069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:50:31.930279 17069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:50:31.933990 17069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:50:31.934213 17069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:50:31.934876 17069 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:50:31.935786 17069 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:50:31.935834 17069 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:31.935874 17069 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:50:31.935890 17069 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:31.945854 17069 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46373
I20260619 14:50:31.945940 17198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46373 every 8 connection(s)
I20260619 14:50:31.946975 17069 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-1/data/info.pb
I20260619 14:50:31.951339 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 17069
I20260619 14:50:31.951805 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:39361
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:39877
--webserver_port=44599
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32973
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:50:31.951853 17201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:31.949930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39877 (local address 127.0.98.2:49049)
0619 14:50:31.950794 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:31.950960 (+ 166us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":144,"threads_started":1}
W20260619 14:50:31.952479 17199 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39877 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:39877: Client connection negotiation failed: client connection to 127.0.98.62:39877: connect: Connection refused (error 111)
I20260619 14:50:31.953891 17201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:31.953391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39877 (local address 127.0.98.2:55933)
0619 14:50:31.953733 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:31.953814 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260619 14:50:31.954819 17201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:31.954560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39877 (local address 127.0.98.2:56037)
0619 14:50:31.954712 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:31.954761 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
W20260619 14:50:31.955070 17199 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260619 14:50:32.095032 17202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:50:32.095510 17202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:50:32.095619 17202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:50:32.105979 17202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:50:32.106104 17202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:50:32.106258 17202 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 14:50:32.120981 17202 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32973
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:39361
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=44599
--tserver_master_addrs=127.0.98.62:39877
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17202
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:50:32.122556 17202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:50:32.124048 17202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:50:32.136992 17208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:32.136978 17210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:32.136978 17207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:50:32.137641 17202 server_base.cc:1034] running on GCE node
I20260619 14:50:32.138216 17202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:50:32.139164 17202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:50:32.140374 17202 hybrid_clock.cc:648] HybridClock initialized: now 1781880632140311 us; error 77 us; skew 500 ppm
Jun 19 14:50:32 dist-test-slave-2sb1 krb5kdc[15746](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880632, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:50:32.144333 17202 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 14:50:32.145435 17202 webserver.cc:460] Webserver started at http://127.0.98.3:44599/ using document root <none> and password file <none>
I20260619 14:50:32.146039 17202 fs_manager.cc:359] Metadata directory not provided
I20260619 14:50:32.146113 17202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:50:32.150909 17202 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.004s
I20260619 14:50:32.153614 17217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:32.155268 17202 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:50:32.155435 17202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-2/wal
uuid: "c25b2be704c74c89a191aa1977240d13"
format_stamp: "Formatted at 2026-06-19 14:49:49 on dist-test-slave-2sb1"
server_key: "a78f727aa1c3c1a6f0023647516cf87a"
server_key_iv: "809fe72fe93ce805a089cdda04a1fc8c"
server_key_version: "encryptionkey@0"
I20260619 14:50:32.156036 17202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:50:32.166563 17202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:50:32.170372 17202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:50:32.170593 17202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:50:32.171214 17202 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:50:32.172050 17202 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:50:32.172129 17202 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:32.172196 17202 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:50:32.172246 17202 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:32.184118 17202 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39361
I20260619 14:50:32.184181 17330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39361 every 8 connection(s)
I20260619 14:50:32.185361 17202 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781880383132139-392-0/minicluster-data/ts-2/data/info.pb
I20260619 14:50:32.190867 17333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:32.188698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39877 (local address 127.0.98.3:46109)
0619 14:50:32.189763 (+ 1065us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:32.189892 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":830,"thread_start_us":186,"threads_started":1}
I20260619 14:50:32.191089 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 17202
W20260619 14:50:32.191570 17331 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:39877 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:39877: Client connection negotiation failed: client connection to 127.0.98.62:39877: connect: Connection refused (error 111)
I20260619 14:50:32.192349 17333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:32.192080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39877 (local address 127.0.98.3:35785)
0619 14:50:32.192192 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:32.192251 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260619 14:50:32.193149 17333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:32.192872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39877 (local address 127.0.98.3:47269)
0619 14:50:32.193012 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:32.193061 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260619 14:50:32.193418 17331 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260619 14:50:32.784780 17335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:32.784176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39877 (local address 127.0.98.1:55425)
0619 14:50:32.784508 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:32.784584 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":99,"threads_started":1}
I20260619 14:50:32.956387 17336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:32.955836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39877 (local address 127.0.98.2:54447)
0619 14:50:32.956197 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:32.956262 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":78,"threads_started":1}
I20260619 14:50:33.194761 17337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:33.194160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39877 (local address 127.0.98.3:55047)
0619 14:50:33.194462 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:33.194633 (+ 171us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":118,"threads_started":1}
I20260619 14:50:33.467300 17338 raft_consensus.cc:482] T 4ecc7db119b642069c5f537791704632 P 4bf51bda973949fcb3568cb66538fcdd [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:50:33.467502 17338 raft_consensus.cc:504] T 4ecc7db119b642069c5f537791704632 P 4bf51bda973949fcb3568cb66538fcdd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9dc81733a644dbe85d11ba50f1f4ebc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46373 } } peers { permanent_uuid: "c25b2be704c74c89a191aa1977240d13" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39361 } } peers { permanent_uuid: "4bf51bda973949fcb3568cb66538fcdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43063 } }
I20260619 14:50:33.468838 17338 leader_election.cc:288] T 4ecc7db119b642069c5f537791704632 P 4bf51bda973949fcb3568cb66538fcdd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b9dc81733a644dbe85d11ba50f1f4ebc (127.0.98.2:46373), c25b2be704c74c89a191aa1977240d13 (127.0.98.3:39361)
Jun 19 14:50:33 dist-test-slave-2sb1 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880631, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Jun 19 14:50:33 dist-test-slave-2sb1 krb5kdc[15746](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880631, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260619 14:50:33.489217 17342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:33.469464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46887 (local address 127.0.98.2:46373)
0619 14:50:33.469829 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:33.469846 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:33.469891 (+ 45us) server_negotiation.cc:407] Connection header received
0619 14:50:33.470800 (+ 909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:33.470808 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:33.471272 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:33.471610 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:33.473820 (+ 2210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:33.475570 (+ 1750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:33.477969 (+ 2399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:33.478891 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:33.479038 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:33.483189 (+ 4151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:33.483194 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:33.483205 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:33.483238 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:33.485287 (+ 2049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:33.486337 (+ 1050us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:33.486345 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:33.486351 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:33.486536 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:33.487050 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:33.487054 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:33.487056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:33.487755 (+ 699us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:33.487879 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:33.488641 (+ 762us) server_negotiation.cc:299] Negotiation successful
0619 14:50:33.489020 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":75,"threads_started":1}
I20260619 14:50:33.489217 17341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:33.469277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48377 (local address 127.0.98.3:39361)
0619 14:50:33.469829 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:33.469850 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:33.469891 (+ 41us) server_negotiation.cc:407] Connection header received
0619 14:50:33.470800 (+ 909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:33.470808 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:33.471272 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:33.471610 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:33.473698 (+ 2088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:33.475570 (+ 1872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:33.477969 (+ 2399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:33.478778 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:33.478996 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:33.482930 (+ 3934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:33.482938 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:33.482957 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:33.482998 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:33.485605 (+ 2607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:33.486337 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:33.486345 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:33.486351 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:33.486536 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:33.487050 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:33.487054 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:33.487056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:33.487650 (+ 594us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:33.487807 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:33.488641 (+ 834us) server_negotiation.cc:299] Negotiation successful
0619 14:50:33.489013 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":197,"threads_started":1}
I20260619 14:50:33.489219 17339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:33.469038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46373 (local address 127.0.98.1:46887)
0619 14:50:33.469375 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:33.469394 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:33.469511 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:33.471765 (+ 2254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:33.471772 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:33.472451 (+ 679us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:33.473394 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:33.473418 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:33.475802 (+ 2384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:33.475807 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:33.477778 (+ 1971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:33.477791 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:33.479012 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:33.479015 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:33.479372 (+ 357us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:33.480357 (+ 985us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:33.480385 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:33.482939 (+ 2554us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:50:33.485450 (+ 2511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:33.485460 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:33.485478 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:33.486195 (+ 717us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:33.486661 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:33.486665 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:33.486667 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:33.486925 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:33.487870 (+ 945us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:50:33.487874 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:50:33.488216 (+ 342us) client_negotiation.cc:769] Sending connection context
0619 14:50:33.488457 (+ 241us) client_negotiation.cc:240] Negotiation successful
0619 14:50:33.489022 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"mutex_wait_us":703,"spinlock_wait_cycles":93440,"thread_start_us":78,"threads_started":1}
I20260619 14:50:33.489219 17340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:33.469114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39361 (local address 127.0.98.1:48377)
0619 14:50:33.469449 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:33.469463 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:33.469731 (+ 268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:33.471765 (+ 2034us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:33.471772 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:33.472458 (+ 686us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:33.473394 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:33.473418 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:33.475802 (+ 2384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:33.475807 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:33.477775 (+ 1968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:33.477791 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:33.478990 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:33.478996 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:33.479372 (+ 376us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:33.480357 (+ 985us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:33.480385 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:33.482626 (+ 2241us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:50:33.485770 (+ 3144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:33.485773 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:33.485775 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:33.486195 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:33.486661 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:33.486665 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:33.486667 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:33.486925 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:33.487791 (+ 866us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:50:33.487797 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:50:33.488214 (+ 417us) client_negotiation.cc:769] Sending connection context
0619 14:50:33.488457 (+ 243us) client_negotiation.cc:240] Negotiation successful
0619 14:50:33.488955 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"mutex_wait_us":457,"spinlock_wait_cycles":20480,"thread_start_us":81,"threads_started":1}
W20260619 14:50:33.492125 16952 leader_election.cc:341] T 4ecc7db119b642069c5f537791704632 P 4bf51bda973949fcb3568cb66538fcdd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c25b2be704c74c89a191aa1977240d13 (127.0.98.3:39361): Not found: Tablet not found: 4ecc7db119b642069c5f537791704632
W20260619 14:50:33.492344 16954 leader_election.cc:341] T 4ecc7db119b642069c5f537791704632 P 4bf51bda973949fcb3568cb66538fcdd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b9dc81733a644dbe85d11ba50f1f4ebc (127.0.98.2:46373): Not found: Tablet not found: 4ecc7db119b642069c5f537791704632
I20260619 14:50:33.492473 16954 leader_election.cc:302] T 4ecc7db119b642069c5f537791704632 P 4bf51bda973949fcb3568cb66538fcdd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bf51bda973949fcb3568cb66538fcdd; no voters: b9dc81733a644dbe85d11ba50f1f4ebc, c25b2be704c74c89a191aa1977240d13
I20260619 14:50:33.493029 17338 raft_consensus.cc:2726] T 4ecc7db119b642069c5f537791704632 P 4bf51bda973949fcb3568cb66538fcdd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 14:50:33.786115 17339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:33.785796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39877 (local address 127.0.98.1:50525)
0619 14:50:33.785950 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:33.786020 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260619 14:50:33.957696 17343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:33.957241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39877 (local address 127.0.98.2:56529)
0619 14:50:33.957516 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:33.957576 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":81,"threads_started":1}
I20260619 14:50:34.196110 17344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:34.195681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39877 (local address 127.0.98.3:36735)
0619 14:50:34.195945 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:34.196010 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":81,"threads_started":1}
I20260619 14:50:34.787624 17345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:34.787152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39877 (local address 127.0.98.1:39431)
0619 14:50:34.787469 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:34.787523 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":141,"threads_started":1}
I20260619 14:50:34.959067 17346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:34.958644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39877 (local address 127.0.98.2:56277)
0619 14:50:34.958908 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:34.958971 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":80,"threads_started":1}
I20260619 14:50:35.197763 17347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:35.197185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39877 (local address 127.0.98.3:59547)
0619 14:50:35.197552 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:35.197644 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":128,"threads_started":1}
I20260619 14:50:35.219882 17348 raft_consensus.cc:482] T 4ecc7db119b642069c5f537791704632 P 4bf51bda973949fcb3568cb66538fcdd [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:50:35.220039 17348 raft_consensus.cc:504] T 4ecc7db119b642069c5f537791704632 P 4bf51bda973949fcb3568cb66538fcdd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9dc81733a644dbe85d11ba50f1f4ebc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46373 } } peers { permanent_uuid: "c25b2be704c74c89a191aa1977240d13" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39361 } } peers { permanent_uuid: "4bf51bda973949fcb3568cb66538fcdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43063 } }
I20260619 14:50:35.220563 17348 leader_election.cc:288] T 4ecc7db119b642069c5f537791704632 P 4bf51bda973949fcb3568cb66538fcdd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b9dc81733a644dbe85d11ba50f1f4ebc (127.0.98.2:46373), c25b2be704c74c89a191aa1977240d13 (127.0.98.3:39361)
W20260619 14:50:35.221693 16952 leader_election.cc:341] T 4ecc7db119b642069c5f537791704632 P 4bf51bda973949fcb3568cb66538fcdd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c25b2be704c74c89a191aa1977240d13 (127.0.98.3:39361): Not found: Tablet not found: 4ecc7db119b642069c5f537791704632
W20260619 14:50:35.221802 16954 leader_election.cc:341] T 4ecc7db119b642069c5f537791704632 P 4bf51bda973949fcb3568cb66538fcdd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b9dc81733a644dbe85d11ba50f1f4ebc (127.0.98.2:46373): Not found: Tablet not found: 4ecc7db119b642069c5f537791704632
I20260619 14:50:35.221910 16954 leader_election.cc:302] T 4ecc7db119b642069c5f537791704632 P 4bf51bda973949fcb3568cb66538fcdd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bf51bda973949fcb3568cb66538fcdd; no voters: b9dc81733a644dbe85d11ba50f1f4ebc, c25b2be704c74c89a191aa1977240d13
I20260619 14:50:35.222085 17348 raft_consensus.cc:2726] T 4ecc7db119b642069c5f537791704632 P 4bf51bda973949fcb3568cb66538fcdd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 14:50:35.789099 17349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:35.788563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39877 (local address 127.0.98.1:48337)
0619 14:50:35.788920 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:35.788988 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":157,"threads_started":1}
I20260619 14:50:35.960491 17350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:35.959942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39877 (local address 127.0.98.2:34017)
0619 14:50:35.960237 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:35.960297 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":78,"threads_started":1}
I20260619 14:50:36.199226 17351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:36.198694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39877 (local address 127.0.98.3:35597)
0619 14:50:36.199023 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:36.199125 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":93,"threads_started":1}
I20260619 14:50:36.790886 17352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:36.790232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39877 (local address 127.0.98.1:50209)
0619 14:50:36.790698 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:36.790779 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":85,"threads_started":1}
I20260619 14:50:36.961951 17353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:36.961464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39877 (local address 127.0.98.2:49483)
0619 14:50:36.961781 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:36.961851 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":82,"threads_started":1}
I20260619 14:50:37.200935 17354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:37.200276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39877 (local address 127.0.98.3:36031)
0619 14:50:37.200762 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:37.200836 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":79,"threads_started":1}
I20260619 14:50:37.792522 17355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:37.792027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39877 (local address 127.0.98.1:54345)
0619 14:50:37.792349 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:37.792414 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":88,"threads_started":1}
I20260619 14:50:37.853005 17356 raft_consensus.cc:482] T 4ecc7db119b642069c5f537791704632 P 4bf51bda973949fcb3568cb66538fcdd [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:50:37.853147 17356 raft_consensus.cc:504] T 4ecc7db119b642069c5f537791704632 P 4bf51bda973949fcb3568cb66538fcdd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9dc81733a644dbe85d11ba50f1f4ebc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46373 } } peers { permanent_uuid: "c25b2be704c74c89a191aa1977240d13" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39361 } } peers { permanent_uuid: "4bf51bda973949fcb3568cb66538fcdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43063 } }
I20260619 14:50:37.853708 17356 leader_election.cc:288] T 4ecc7db119b642069c5f537791704632 P 4bf51bda973949fcb3568cb66538fcdd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b9dc81733a644dbe85d11ba50f1f4ebc (127.0.98.2:46373), c25b2be704c74c89a191aa1977240d13 (127.0.98.3:39361)
W20260619 14:50:37.854598 16954 leader_election.cc:341] T 4ecc7db119b642069c5f537791704632 P 4bf51bda973949fcb3568cb66538fcdd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b9dc81733a644dbe85d11ba50f1f4ebc (127.0.98.2:46373): Not found: Tablet not found: 4ecc7db119b642069c5f537791704632
W20260619 14:50:37.854712 16952 leader_election.cc:341] T 4ecc7db119b642069c5f537791704632 P 4bf51bda973949fcb3568cb66538fcdd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c25b2be704c74c89a191aa1977240d13 (127.0.98.3:39361): Not found: Tablet not found: 4ecc7db119b642069c5f537791704632
I20260619 14:50:37.854830 16952 leader_election.cc:302] T 4ecc7db119b642069c5f537791704632 P 4bf51bda973949fcb3568cb66538fcdd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bf51bda973949fcb3568cb66538fcdd; no voters: b9dc81733a644dbe85d11ba50f1f4ebc, c25b2be704c74c89a191aa1977240d13
I20260619 14:50:37.855028 17356 raft_consensus.cc:2726] T 4ecc7db119b642069c5f537791704632 P 4bf51bda973949fcb3568cb66538fcdd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 14:50:37.963647 17357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:37.963056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39877 (local address 127.0.98.2:45177)
0619 14:50:37.963361 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:37.963425 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":100,"threads_started":1}
I20260619 14:50:38.202507 17358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:38.201984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39877 (local address 127.0.98.3:52125)
0619 14:50:38.202347 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:38.202407 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":163,"threads_started":1}
I20260619 14:50:38.793895 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:38.793438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39877 (local address 127.0.98.1:60863)
0619 14:50:38.793732 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:38.793796 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":89,"threads_started":1}
I20260619 14:50:38.965245 17360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:38.964737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39877 (local address 127.0.98.2:54975)
0619 14:50:38.965091 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:38.965148 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":92,"threads_started":1}
I20260619 14:50:39.204042 17361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:39.203523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39877 (local address 127.0.98.3:44235)
0619 14:50:39.203861 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:39.203932 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":82,"threads_started":1}
I20260619 14:50:39.480331 17362 raft_consensus.cc:482] T 4ecc7db119b642069c5f537791704632 P 4bf51bda973949fcb3568cb66538fcdd [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:50:39.480633 17362 raft_consensus.cc:504] T 4ecc7db119b642069c5f537791704632 P 4bf51bda973949fcb3568cb66538fcdd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9dc81733a644dbe85d11ba50f1f4ebc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46373 } } peers { permanent_uuid: "c25b2be704c74c89a191aa1977240d13" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39361 } } peers { permanent_uuid: "4bf51bda973949fcb3568cb66538fcdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43063 } }
I20260619 14:50:39.481256 17362 leader_election.cc:288] T 4ecc7db119b642069c5f537791704632 P 4bf51bda973949fcb3568cb66538fcdd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b9dc81733a644dbe85d11ba50f1f4ebc (127.0.98.2:46373), c25b2be704c74c89a191aa1977240d13 (127.0.98.3:39361)
W20260619 14:50:39.482149 16954 leader_election.cc:341] T 4ecc7db119b642069c5f537791704632 P 4bf51bda973949fcb3568cb66538fcdd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b9dc81733a644dbe85d11ba50f1f4ebc (127.0.98.2:46373): Not found: Tablet not found: 4ecc7db119b642069c5f537791704632
W20260619 14:50:39.482343 16952 leader_election.cc:341] T 4ecc7db119b642069c5f537791704632 P 4bf51bda973949fcb3568cb66538fcdd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c25b2be704c74c89a191aa1977240d13 (127.0.98.3:39361): Not found: Tablet not found: 4ecc7db119b642069c5f537791704632
I20260619 14:50:39.482419 16952 leader_election.cc:302] T 4ecc7db119b642069c5f537791704632 P 4bf51bda973949fcb3568cb66538fcdd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bf51bda973949fcb3568cb66538fcdd; no voters: b9dc81733a644dbe85d11ba50f1f4ebc, c25b2be704c74c89a191aa1977240d13
I20260619 14:50:39.482673 17362 raft_consensus.cc:2726] T 4ecc7db119b642069c5f537791704632 P 4bf51bda973949fcb3568cb66538fcdd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 14:50:39.795341 17363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:39.794879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39877 (local address 127.0.98.1:45621)
0619 14:50:39.795164 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:39.795233 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":83,"threads_started":1}
I20260619 14:50:39.966773 17364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:39.966219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39877 (local address 127.0.98.2:44907)
0619 14:50:39.966602 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:39.966671 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":110,"threads_started":1}
I20260619 14:50:40.205674 17365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:40.205177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39877 (local address 127.0.98.3:46619)
0619 14:50:40.205476 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:40.205556 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":90,"threads_started":1}
I20260619 14:50:40.796980 17366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:40.796401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39877 (local address 127.0.98.1:56289)
0619 14:50:40.796801 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:40.796880 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":117,"threads_started":1}
I20260619 14:50:40.968924 17367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:40.968194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39877 (local address 127.0.98.2:60051)
0619 14:50:40.968697 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:40.968789 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":109,"threads_started":1}
I20260619 14:50:41.026805 17368 raft_consensus.cc:482] T 4ecc7db119b642069c5f537791704632 P 4bf51bda973949fcb3568cb66538fcdd [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:50:41.026952 17368 raft_consensus.cc:504] T 4ecc7db119b642069c5f537791704632 P 4bf51bda973949fcb3568cb66538fcdd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b9dc81733a644dbe85d11ba50f1f4ebc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46373 } } peers { permanent_uuid: "c25b2be704c74c89a191aa1977240d13" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39361 } } peers { permanent_uuid: "4bf51bda973949fcb3568cb66538fcdd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43063 } }
I20260619 14:50:41.027380 17368 leader_election.cc:288] T 4ecc7db119b642069c5f537791704632 P 4bf51bda973949fcb3568cb66538fcdd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b9dc81733a644dbe85d11ba50f1f4ebc (127.0.98.2:46373), c25b2be704c74c89a191aa1977240d13 (127.0.98.3:39361)
W20260619 14:50:41.028344 16954 leader_election.cc:341] T 4ecc7db119b642069c5f537791704632 P 4bf51bda973949fcb3568cb66538fcdd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b9dc81733a644dbe85d11ba50f1f4ebc (127.0.98.2:46373): Not found: Tablet not found: 4ecc7db119b642069c5f537791704632
W20260619 14:50:41.028458 16952 leader_election.cc:341] T 4ecc7db119b642069c5f537791704632 P 4bf51bda973949fcb3568cb66538fcdd [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c25b2be704c74c89a191aa1977240d13 (127.0.98.3:39361): Not found: Tablet not found: 4ecc7db119b642069c5f537791704632
I20260619 14:50:41.028533 16952 leader_election.cc:302] T 4ecc7db119b642069c5f537791704632 P 4bf51bda973949fcb3568cb66538fcdd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4bf51bda973949fcb3568cb66538fcdd; no voters: b9dc81733a644dbe85d11ba50f1f4ebc, c25b2be704c74c89a191aa1977240d13
I20260619 14:50:41.028753 17368 raft_consensus.cc:2726] T 4ecc7db119b642069c5f537791704632 P 4bf51bda973949fcb3568cb66538fcdd [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260619 14:50:41.207054 17369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:41.206576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39877 (local address 127.0.98.3:41709)
0619 14:50:41.206891 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:41.206959 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":92,"threads_started":1}
I20260619 14:50:41.798636 17370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:41.798130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39877 (local address 127.0.98.1:55267)
0619 14:50:41.798446 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:41.798520 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":100,"threads_started":1}
I20260619 14:50:41.970794 17371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:41.970270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39877 (local address 127.0.98.2:41861)
0619 14:50:41.970583 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:41.970667 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":110,"threads_started":1}
I20260619 14:50:42.208523 17372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:42.208052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39877 (local address 127.0.98.3:49159)
0619 14:50:42.208339 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:42.208405 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":80,"threads_started":1}
I20260619 14:50:42.800154 17373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:42.799700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39877 (local address 127.0.98.1:51761)
0619 14:50:42.799973 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:42.800034 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":79,"threads_started":1}
I20260619 14:50:42.972249 17374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:42.971717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39877 (local address 127.0.98.2:50157)
0619 14:50:42.972011 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:42.972083 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":76,"threads_started":1}
I20260619 14:50:43.210064 17375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:43.209553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39877 (local address 127.0.98.3:41365)
0619 14:50:43.209870 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:43.209936 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":104,"threads_started":1}
I20260619 14:50:43.801787 17376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:43.801197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39877 (local address 127.0.98.1:58123)
0619 14:50:43.801524 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:43.801601 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":112,"threads_started":1}
I20260619 14:50:43.973579 17377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:43.973128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39877 (local address 127.0.98.2:47621)
0619 14:50:43.973390 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:43.973453 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":99,"threads_started":1}
I20260619 14:50:44.211457 17378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:44.210965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39877 (local address 127.0.98.3:55679)
0619 14:50:44.211262 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:44.211330 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":95,"threads_started":1}
I20260619 14:50:44.803172 17379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:44.802706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39877 (local address 127.0.98.1:58665)
0619 14:50:44.803020 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:44.803078 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":114,"threads_started":1}
I20260619 14:50:44.974916 17380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:44.974435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39877 (local address 127.0.98.2:44611)
0619 14:50:44.974754 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:44.974818 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":121,"threads_started":1}
I20260619 14:50:45.213033 17381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:45.212479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39877 (local address 127.0.98.3:54497)
0619 14:50:45.212858 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:45.212936 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":97,"threads_started":1}
I20260619 14:50:45.804538 17382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:45.804091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39877 (local address 127.0.98.1:44299)
0619 14:50:45.804364 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:45.804430 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":80,"threads_started":1}
I20260619 14:50:45.976431 17383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:45.975923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39877 (local address 127.0.98.2:54047)
0619 14:50:45.976254 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:45.976325 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":102,"threads_started":1}
I20260619 14:50:46.214422 17384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:46.213907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39877 (local address 127.0.98.3:45155)
0619 14:50:46.214235 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:46.214314 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":169,"threads_started":1}
I20260619 14:50:46.806207 17385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:46.805711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39877 (local address 127.0.98.1:33025)
0619 14:50:46.806035 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:46.806112 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":80,"threads_started":1}
I20260619 14:50:46.977875 17386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:46.977438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39877 (local address 127.0.98.2:41231)
0619 14:50:46.977716 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:46.977776 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:39877: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":84,"threads_started":1}
I20260619 14:50:47.200295 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 16895
I20260619 14:50:47.206269 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 17069
I20260619 14:50:47.212064 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 17202
2026-06-19T14:50:47Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59863 ms)
[----------] 30 tests from SecurityITest (264066 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880383132139-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:50:47 dist-test-slave-2sb1 krb5kdc[17389](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:50:47 dist-test-slave-2sb1 krb5kdc[17389](info): set up 2 sockets
Jun 19 14:50:47 dist-test-slave-2sb1 krb5kdc[17389](info): commencing operation
krb5kdc: starting...
W20260619 14:50:48.324915 392 mini_kdc.cc:121] Time spent starting KDC: real 1.089s user 0.007s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880383132139-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880383132139-392-0/krb5kdc/test-user.keytab.
Jun 19 14:50:48 dist-test-slave-2sb1 krb5kdc[17389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880648, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:50:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:50:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 14:50:48.478278 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880383132139-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880383132139-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880383132139-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34323
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34323
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:50:48.598908 17402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:50:48.599344 17402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:50:48.599447 17402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:50:48.604403 17402 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:50:48.604573 17402 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:50:48.604620 17402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:50:48.604725 17402 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:50:48.604782 17402 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:50:48.611320 17402 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880383132139-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880383132139-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34323
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34323
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17402
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880383132139-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:50:48.612679 17402 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:50:48.613929 17402 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 14:50:48.625491 17402 server_base.cc:1034] running on GCE node
W20260619 14:50:48.625506 17410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:48.625385 17407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:48.625389 17408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:50:48.626384 17402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:50:48.627818 17402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:50:48.629083 17402 hybrid_clock.cc:648] HybridClock initialized: now 1781880648629058 us; error 46 us; skew 500 ppm
Jun 19 14:50:48 dist-test-slave-2sb1 krb5kdc[17389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880648, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:50:48.632639 17402 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 14:50:48.633984 17402 webserver.cc:460] Webserver started at http://127.0.98.62:32881/ using document root <none> and password file <none>
I20260619 14:50:48.634613 17402 fs_manager.cc:359] Metadata directory not provided
I20260619 14:50:48.634713 17402 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:50:48.634963 17402 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:50:48.637279 17402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880383132139-392-0/minicluster-data/master-0/data/instance:
uuid: "7bb2489393334cfbaad62db8963ca9f0"
format_stamp: "Formatted at 2026-06-19 14:50:48 on dist-test-slave-2sb1"
server_key: "4c86066c8896ac8f2482dab102908a6a"
server_key_iv: "01b8972e17ee01518b318b0783b5d690"
server_key_version: "encryptionkey@0"
I20260619 14:50:48.637912 17402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880383132139-392-0/minicluster-data/master-0/wal/instance:
uuid: "7bb2489393334cfbaad62db8963ca9f0"
format_stamp: "Formatted at 2026-06-19 14:50:48 on dist-test-slave-2sb1"
server_key: "4c86066c8896ac8f2482dab102908a6a"
server_key_iv: "01b8972e17ee01518b318b0783b5d690"
server_key_version: "encryptionkey@0"
I20260619 14:50:48.642280 17402 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 14:50:48.645452 17417 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:48.647291 17402 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:50:48.647512 17402 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880383132139-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880383132139-392-0/minicluster-data/master-0/wal
uuid: "7bb2489393334cfbaad62db8963ca9f0"
format_stamp: "Formatted at 2026-06-19 14:50:48 on dist-test-slave-2sb1"
server_key: "4c86066c8896ac8f2482dab102908a6a"
server_key_iv: "01b8972e17ee01518b318b0783b5d690"
server_key_version: "encryptionkey@0"
I20260619 14:50:48.647682 17402 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880383132139-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880383132139-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880383132139-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:50:48.678469 17402 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:50:48.683002 17402 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:50:48.683269 17402 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:50:48.691352 17402 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34323
I20260619 14:50:48.691376 17469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34323 every 8 connection(s)
I20260619 14:50:48.692458 17402 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:50:48.695459 17470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:50:48.696539 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 17402
I20260619 14:50:48.696703 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880383132139-392-0/minicluster-data/master-0/wal/instance
I20260619 14:50:48.697022 392 external_mini_cluster.cc:1428] Setting key 66ac2c46a2bc86a50ea8f09b28baa040
I20260619 14:50:48.702344 17470 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bb2489393334cfbaad62db8963ca9f0: Bootstrap starting.
I20260619 14:50:48.705329 17470 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7bb2489393334cfbaad62db8963ca9f0: Neither blocks nor log segments found. Creating new log.
I20260619 14:50:48.706146 17470 log.cc:824] T 00000000000000000000000000000000 P 7bb2489393334cfbaad62db8963ca9f0: Log is configured to *not* fsync() on all Append() calls
Jun 19 14:50:48 dist-test-slave-2sb1 krb5kdc[17389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880648, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:50:48.708810 17470 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7bb2489393334cfbaad62db8963ca9f0: No bootstrap required, opened a new log
I20260619 14:50:48.713889 17470 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7bb2489393334cfbaad62db8963ca9f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bb2489393334cfbaad62db8963ca9f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34323 } }
I20260619 14:50:48.714283 17470 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7bb2489393334cfbaad62db8963ca9f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:50:48.714339 17470 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7bb2489393334cfbaad62db8963ca9f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7bb2489393334cfbaad62db8963ca9f0, State: Initialized, Role: FOLLOWER
I20260619 14:50:48.714876 17470 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7bb2489393334cfbaad62db8963ca9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bb2489393334cfbaad62db8963ca9f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34323 } }
I20260619 14:50:48.715052 17470 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7bb2489393334cfbaad62db8963ca9f0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:50:48.715112 17470 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7bb2489393334cfbaad62db8963ca9f0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:50:48.715201 17470 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7bb2489393334cfbaad62db8963ca9f0 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:50:48.715556 17473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:48.698671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34122 (local address 127.0.98.62:34323)
0619 14:50:48.699242 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:48.699259 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:48.699299 (+ 40us) server_negotiation.cc:407] Connection header received
0619 14:50:48.700167 (+ 868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:48.700189 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:48.700595 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:48.701108 (+ 513us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:48.702088 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:48.703409 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:48.704851 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:48.705636 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:48.705832 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:48.709250 (+ 3418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:48.709258 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:48.709276 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:48.709312 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:48.711579 (+ 2267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:48.712264 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:48.712271 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:48.712279 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:48.712435 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:48.713023 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:48.713027 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:48.713029 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:48.713618 (+ 589us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:48.713845 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:48.714150 (+ 305us) server_negotiation.cc:299] Negotiation successful
0619 14:50:48.714472 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":141,"threads_started":1}
I20260619 14:50:48.716820 17470 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7bb2489393334cfbaad62db8963ca9f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bb2489393334cfbaad62db8963ca9f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34323 } }
I20260619 14:50:48.718356 17470 leader_election.cc:302] T 00000000000000000000000000000000 P 7bb2489393334cfbaad62db8963ca9f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7bb2489393334cfbaad62db8963ca9f0; no voters:
I20260619 14:50:48.718762 17470 leader_election.cc:288] T 00000000000000000000000000000000 P 7bb2489393334cfbaad62db8963ca9f0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:50:48.718928 17475 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7bb2489393334cfbaad62db8963ca9f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:50:48.719206 17475 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7bb2489393334cfbaad62db8963ca9f0 [term 1 LEADER]: Becoming Leader. State: Replica: 7bb2489393334cfbaad62db8963ca9f0, State: Running, Role: LEADER
I20260619 14:50:48.719671 17475 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7bb2489393334cfbaad62db8963ca9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bb2489393334cfbaad62db8963ca9f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34323 } }
I20260619 14:50:48.720099 17470 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7bb2489393334cfbaad62db8963ca9f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:50:48.721560 17477 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bb2489393334cfbaad62db8963ca9f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7bb2489393334cfbaad62db8963ca9f0. Latest consensus state: current_term: 1 leader_uuid: "7bb2489393334cfbaad62db8963ca9f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bb2489393334cfbaad62db8963ca9f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34323 } } }
I20260619 14:50:48.721700 17477 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bb2489393334cfbaad62db8963ca9f0 [sys.catalog]: This master's current role is: LEADER
I20260619 14:50:48.722038 17476 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7bb2489393334cfbaad62db8963ca9f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7bb2489393334cfbaad62db8963ca9f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7bb2489393334cfbaad62db8963ca9f0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34323 } } }
I20260619 14:50:48.722146 17476 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7bb2489393334cfbaad62db8963ca9f0 [sys.catalog]: This master's current role is: LEADER
I20260619 14:50:48.722465 17485 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:50:48.726063 17485 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:50:48.731983 17485 catalog_manager.cc:1347] Generated new cluster ID: 7aae9c82ee7d4c1b9e3b57c01e46fc89
I20260619 14:50:48.732071 17485 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:50:48.749646 17485 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:50:48.751153 17485 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:50:48.757371 17485 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7bb2489393334cfbaad62db8963ca9f0: Generated new TSK 0
I20260619 14:50:48.758289 17485 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880383132139-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880383132139-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 14:50:48.805049 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880383132139-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880383132139-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880383132139-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880383132139-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34323
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880383132139-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:50:48.941075 17496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:50:48.941422 17496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:50:48.941516 17496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:50:48.946094 17496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:50:48.946215 17496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:50:48.946312 17496 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 14:50:48.952279 17496 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880383132139-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880383132139-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880383132139-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880383132139-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34323
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17496
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880383132139-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:50:48.953557 17496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:50:48.954656 17496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:50:48.965314 17501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:48.965281 17504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:48.965355 17502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:50:48.965736 17496 server_base.cc:1034] running on GCE node
I20260619 14:50:48.966281 17496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:50:48.967022 17496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:50:48.968245 17496 hybrid_clock.cc:648] HybridClock initialized: now 1781880648968208 us; error 50 us; skew 500 ppm
Jun 19 14:50:48 dist-test-slave-2sb1 krb5kdc[17389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880648, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:50:48.971547 17496 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 14:50:48.972492 17496 webserver.cc:460] Webserver started at http://127.0.98.1:45127/ using document root <none> and password file <none>
I20260619 14:50:48.973169 17496 fs_manager.cc:359] Metadata directory not provided
I20260619 14:50:48.973238 17496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:50:48.973501 17496 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:50:48.975719 17496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880383132139-392-0/minicluster-data/ts-0/data/instance:
uuid: "02fc9904da434210a6a7fda483348222"
format_stamp: "Formatted at 2026-06-19 14:50:48 on dist-test-slave-2sb1"
server_key: "1279d82b9be4cd937c7cf7ed1abe5cd2"
server_key_iv: "6445ae6476585644857df8867dd76b21"
server_key_version: "encryptionkey@0"
I20260619 14:50:48.976346 17496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880383132139-392-0/minicluster-data/ts-0/wal/instance:
uuid: "02fc9904da434210a6a7fda483348222"
format_stamp: "Formatted at 2026-06-19 14:50:48 on dist-test-slave-2sb1"
server_key: "1279d82b9be4cd937c7cf7ed1abe5cd2"
server_key_iv: "6445ae6476585644857df8867dd76b21"
server_key_version: "encryptionkey@0"
I20260619 14:50:48.980512 17496 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260619 14:50:48.983538 17511 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:48.985138 17496 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:50:48.985265 17496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880383132139-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880383132139-392-0/minicluster-data/ts-0/wal
uuid: "02fc9904da434210a6a7fda483348222"
format_stamp: "Formatted at 2026-06-19 14:50:48 on dist-test-slave-2sb1"
server_key: "1279d82b9be4cd937c7cf7ed1abe5cd2"
server_key_iv: "6445ae6476585644857df8867dd76b21"
server_key_version: "encryptionkey@0"
I20260619 14:50:48.985430 17496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880383132139-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880383132139-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880383132139-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:50:49.003615 17496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:50:49.007596 17496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:50:49.007872 17496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:50:49.009017 17496 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:50:49.010273 17496 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:50:49.010332 17496 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:49.010380 17496 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:50:49.010397 17496 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:49.021517 17496 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41973
I20260619 14:50:49.021600 17624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41973 every 8 connection(s)
I20260619 14:50:49.022637 17496 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880383132139-392-0/minicluster-data/ts-0/data/info.pb
I20260619 14:50:49.023553 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 17496
I20260619 14:50:49.023699 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880383132139-392-0/minicluster-data/ts-0/wal/instance
I20260619 14:50:49.024091 392 external_mini_cluster.cc:1428] Setting key 3853f201b1cee7b95656ddc7309476f8
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Jun 19 14:50:49 dist-test-slave-2sb1 krb5kdc[17389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880648, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:50:49.046957 17473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:49.025236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37469 (local address 127.0.98.62:34323)
0619 14:50:49.025399 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:49.025403 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:49.026299 (+ 896us) server_negotiation.cc:407] Connection header received
0619 14:50:49.028426 (+ 2127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:49.028434 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:49.028744 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:49.028864 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:49.030743 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:49.031607 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:49.033804 (+ 2197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:49.034060 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:49.034163 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:49.041239 (+ 7076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:49.041245 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:49.041250 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:49.041296 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:49.043871 (+ 2575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:49.044928 (+ 1057us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:49.044934 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:49.044936 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:49.045054 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:49.045473 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:49.045479 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:49.045483 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:49.045832 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:49.045959 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:49.046584 (+ 625us) server_negotiation.cc:299] Negotiation successful
0619 14:50:49.046757 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 14:50:49.047896 17627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:49.025536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34323 (local address 127.0.98.1:37469)
0619 14:50:49.026112 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:49.026173 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:49.028145 (+ 1972us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:49.029082 (+ 937us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:49.029090 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:49.029765 (+ 675us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:49.030534 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:49.030559 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:49.031776 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:49.031783 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:49.033603 (+ 1820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:49.033624 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:49.034261 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:49.034265 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:49.034514 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:49.035392 (+ 878us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:49.035434 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:49.040906 (+ 5472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:50:49.044106 (+ 3200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:49.044114 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:49.044131 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:49.044782 (+ 651us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:49.045178 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:49.045181 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:49.045182 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:49.045345 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:49.045977 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:50:49.045984 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:50:49.046293 (+ 309us) client_negotiation.cc:769] Sending connection context
0619 14:50:49.046473 (+ 180us) client_negotiation.cc:240] Negotiation successful
0619 14:50:49.046783 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":179,"threads_started":1}
I20260619 14:50:49.049834 17625 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34323
I20260619 14:50:49.050092 17625 heartbeater.cc:464] Registering TS with master...
I20260619 14:50:49.051093 17625 heartbeater.cc:511] Master 127.0.98.62:34323 requested a full tablet report, sending...
I20260619 14:50:49.053591 17430 ts_manager.cc:194] Registered new tserver with Master: 02fc9904da434210a6a7fda483348222 (127.0.98.1:41973)
I20260619 14:50:49.055434 17430 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:37469
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880383132139-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880383132139-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 14:50:49.062860 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880383132139-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880383132139-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880383132139-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880383132139-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34323
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880383132139-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:50:49.185916 17630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:50:49.186237 17630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:50:49.186283 17630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:50:49.190868 17630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:50:49.190948 17630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:50:49.191017 17630 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 14:50:49.196834 17630 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880383132139-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880383132139-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880383132139-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880383132139-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34323
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17630
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880383132139-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:50:49.198024 17630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:50:49.199126 17630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:50:49.210090 17638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:49.210135 17636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:49.210090 17635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:50:49.210616 17630 server_base.cc:1034] running on GCE node
I20260619 14:50:49.211071 17630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:50:49.211772 17630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:50:49.213004 17630 hybrid_clock.cc:648] HybridClock initialized: now 1781880649212985 us; error 33 us; skew 500 ppm
Jun 19 14:50:49 dist-test-slave-2sb1 krb5kdc[17389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880649, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:50:49.216512 17630 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 14:50:49.217658 17630 webserver.cc:460] Webserver started at http://127.0.98.2:35983/ using document root <none> and password file <none>
I20260619 14:50:49.218281 17630 fs_manager.cc:359] Metadata directory not provided
I20260619 14:50:49.218393 17630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:50:49.218681 17630 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:50:49.221100 17630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880383132139-392-0/minicluster-data/ts-1/data/instance:
uuid: "f2250f53fc534800987e77f3c8f5d0e7"
format_stamp: "Formatted at 2026-06-19 14:50:49 on dist-test-slave-2sb1"
server_key: "b07e23cd43825b2fa531d4ede114c6d7"
server_key_iv: "c9e1b0fb7589e0f8037a4a27daa68857"
server_key_version: "encryptionkey@0"
I20260619 14:50:49.221876 17630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880383132139-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f2250f53fc534800987e77f3c8f5d0e7"
format_stamp: "Formatted at 2026-06-19 14:50:49 on dist-test-slave-2sb1"
server_key: "b07e23cd43825b2fa531d4ede114c6d7"
server_key_iv: "c9e1b0fb7589e0f8037a4a27daa68857"
server_key_version: "encryptionkey@0"
I20260619 14:50:49.226325 17630 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 14:50:49.229471 17645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:49.231256 17630 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:50:49.231424 17630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880383132139-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880383132139-392-0/minicluster-data/ts-1/wal
uuid: "f2250f53fc534800987e77f3c8f5d0e7"
format_stamp: "Formatted at 2026-06-19 14:50:49 on dist-test-slave-2sb1"
server_key: "b07e23cd43825b2fa531d4ede114c6d7"
server_key_iv: "c9e1b0fb7589e0f8037a4a27daa68857"
server_key_version: "encryptionkey@0"
I20260619 14:50:49.231573 17630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880383132139-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880383132139-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880383132139-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:50:49.256541 17630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:50:49.260993 17630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:50:49.261264 17630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:50:49.261937 17630 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:50:49.262852 17630 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:50:49.262929 17630 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:49.262998 17630 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:50:49.263049 17630 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:49.277145 17630 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40125
I20260619 14:50:49.277253 17758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40125 every 8 connection(s)
I20260619 14:50:49.278344 17630 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880383132139-392-0/minicluster-data/ts-1/data/info.pb
I20260619 14:50:49.281464 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 17630
I20260619 14:50:49.281776 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880383132139-392-0/minicluster-data/ts-1/wal/instance
I20260619 14:50:49.282162 392 external_mini_cluster.cc:1428] Setting key 9a5409e769a871058f1bfec7cb3eecfd
Jun 19 14:50:49 dist-test-slave-2sb1 krb5kdc[17389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880649, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260619 14:50:49.299350 17473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:49.280740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43253 (local address 127.0.98.62:34323)
0619 14:50:49.280980 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:49.280985 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:49.281976 (+ 991us) server_negotiation.cc:407] Connection header received
0619 14:50:49.283416 (+ 1440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:49.283421 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:49.283506 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:49.283628 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:49.285425 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:49.286435 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:49.288445 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:49.288900 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:49.289058 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:49.292649 (+ 3591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:49.292654 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:49.292658 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:49.292732 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:49.295351 (+ 2619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:49.296259 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:49.296263 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:49.296264 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:49.296371 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:49.297218 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:49.297224 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:49.297229 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:49.297529 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:49.297662 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:49.298754 (+ 1092us) server_negotiation.cc:299] Negotiation successful
0619 14:50:49.299072 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260619 14:50:49.300135 17761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:49.281205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34323 (local address 127.0.98.2:43253)
0619 14:50:49.281761 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:49.281838 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:49.282970 (+ 1132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:49.284145 (+ 1175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:49.284153 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:49.284598 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:49.285236 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:49.285253 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:49.286666 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:49.286672 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:49.288256 (+ 1584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:49.288271 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:49.289121 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:49.289124 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:49.289383 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:49.290265 (+ 882us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:49.290283 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:49.292308 (+ 2025us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:50:49.295765 (+ 3457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:49.295772 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:49.295790 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:49.296127 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:49.296770 (+ 643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:49.296776 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:49.296781 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:49.297070 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:49.297787 (+ 717us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:50:49.297804 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:50:49.298350 (+ 546us) client_negotiation.cc:769] Sending connection context
0619 14:50:49.298665 (+ 315us) client_negotiation.cc:240] Negotiation successful
0619 14:50:49.299122 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":168,"threads_started":1}
I20260619 14:50:49.301967 17759 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34323
I20260619 14:50:49.302285 17759 heartbeater.cc:464] Registering TS with master...
I20260619 14:50:49.303001 17759 heartbeater.cc:511] Master 127.0.98.62:34323 requested a full tablet report, sending...
I20260619 14:50:49.305080 17430 ts_manager.cc:194] Registered new tserver with Master: f2250f53fc534800987e77f3c8f5d0e7 (127.0.98.2:40125)
I20260619 14:50:49.305881 17430 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:43253
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880383132139-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880383132139-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 14:50:49.326494 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880383132139-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880383132139-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880383132139-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880383132139-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34323
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880383132139-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:50:49.457522 17764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:50:49.457881 17764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:50:49.457969 17764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:50:49.462302 17764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:50:49.462430 17764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:50:49.462530 17764 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 14:50:49.468327 17764 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880383132139-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880383132139-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880383132139-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880383132139-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34323
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17764
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880383132139-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:50:49.469556 17764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:50:49.470621 17764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:50:49.481700 17772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:50:49.481753 17764 server_base.cc:1034] running on GCE node
W20260619 14:50:49.481707 17769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:49.481707 17770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:50:49.482581 17764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:50:49.483225 17764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:50:49.484401 17764 hybrid_clock.cc:648] HybridClock initialized: now 1781880649484390 us; error 31 us; skew 500 ppm
Jun 19 14:50:49 dist-test-slave-2sb1 krb5kdc[17389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880649, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:50:49.487674 17764 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 14:50:49.488739 17764 webserver.cc:460] Webserver started at http://127.0.98.3:37133/ using document root <none> and password file <none>
I20260619 14:50:49.489372 17764 fs_manager.cc:359] Metadata directory not provided
I20260619 14:50:49.489437 17764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:50:49.489616 17764 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:50:49.492049 17764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880383132139-392-0/minicluster-data/ts-2/data/instance:
uuid: "0b7e9761278c49b59bc8263c08839b3e"
format_stamp: "Formatted at 2026-06-19 14:50:49 on dist-test-slave-2sb1"
server_key: "85b8e66ae22fe07be63dae5839217d0a"
server_key_iv: "d07dc97186d2d1784bab61a3db44ab22"
server_key_version: "encryptionkey@0"
I20260619 14:50:49.492635 17764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880383132139-392-0/minicluster-data/ts-2/wal/instance:
uuid: "0b7e9761278c49b59bc8263c08839b3e"
format_stamp: "Formatted at 2026-06-19 14:50:49 on dist-test-slave-2sb1"
server_key: "85b8e66ae22fe07be63dae5839217d0a"
server_key_iv: "d07dc97186d2d1784bab61a3db44ab22"
server_key_version: "encryptionkey@0"
I20260619 14:50:49.497270 17764 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 14:50:49.500231 17779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:49.502081 17764 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:50:49.502244 17764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880383132139-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880383132139-392-0/minicluster-data/ts-2/wal
uuid: "0b7e9761278c49b59bc8263c08839b3e"
format_stamp: "Formatted at 2026-06-19 14:50:49 on dist-test-slave-2sb1"
server_key: "85b8e66ae22fe07be63dae5839217d0a"
server_key_iv: "d07dc97186d2d1784bab61a3db44ab22"
server_key_version: "encryptionkey@0"
I20260619 14:50:49.502378 17764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880383132139-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880383132139-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880383132139-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:50:49.522094 17764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:50:49.526062 17764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:50:49.526300 17764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:50:49.526961 17764 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:50:49.527879 17764 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:50:49.527956 17764 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:49.528041 17764 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:50:49.528079 17764 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:49.539057 17764 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37155
I20260619 14:50:49.539103 17892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37155 every 8 connection(s)
I20260619 14:50:49.540074 17764 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880383132139-392-0/minicluster-data/ts-2/data/info.pb
I20260619 14:50:49.545363 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 17764
I20260619 14:50:49.545519 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880383132139-392-0/minicluster-data/ts-2/wal/instance
I20260619 14:50:49.545825 392 external_mini_cluster.cc:1428] Setting key af92cc40c805ca51cc178472130b5720
Jun 19 14:50:49 dist-test-slave-2sb1 krb5kdc[17389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880649, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:50:49.557210 17473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:49.542221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40977 (local address 127.0.98.62:34323)
0619 14:50:49.542386 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:49.542389 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:49.543257 (+ 868us) server_negotiation.cc:407] Connection header received
0619 14:50:49.544740 (+ 1483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:49.544744 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:49.544806 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:49.544920 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:49.546560 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:49.547505 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:49.549130 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:49.549356 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:49.549535 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:49.552844 (+ 3309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:49.552847 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:49.552849 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:49.552883 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:49.554502 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:49.555195 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:49.555198 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:49.555199 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:49.555284 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:49.555780 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:49.555783 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:49.555784 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:49.556100 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:49.556194 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:49.556939 (+ 745us) server_negotiation.cc:299] Negotiation successful
0619 14:50:49.557080 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 14:50:49.558276 17895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:49.542600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34323 (local address 127.0.98.3:40977)
0619 14:50:49.543098 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:49.543143 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:49.544371 (+ 1228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:49.545127 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:49.545139 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:49.545687 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:49.546372 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:49.546395 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:49.547720 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:49.547723 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:49.548970 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:49.548982 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:49.549624 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:49.549627 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:49.549803 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:49.550703 (+ 900us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:49.550721 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:49.552562 (+ 1841us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:50:49.554700 (+ 2138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:49.554706 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:49.554731 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:49.555052 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:49.555415 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:49.555418 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:49.555420 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:49.555612 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:49.556238 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:50:49.556244 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:50:49.556535 (+ 291us) client_negotiation.cc:769] Sending connection context
0619 14:50:49.556835 (+ 300us) client_negotiation.cc:240] Negotiation successful
0619 14:50:49.557155 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":128,"threads_started":1}
I20260619 14:50:49.560101 17893 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34323
I20260619 14:50:49.560433 17893 heartbeater.cc:464] Registering TS with master...
I20260619 14:50:49.561089 17893 heartbeater.cc:511] Master 127.0.98.62:34323 requested a full tablet report, sending...
I20260619 14:50:49.562456 17434 ts_manager.cc:194] Registered new tserver with Master: 0b7e9761278c49b59bc8263c08839b3e (127.0.98.3:37155)
I20260619 14:50:49.563199 17434 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40977
I20260619 14:50:49.573501 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:50:49.585074 17473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:49.575979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43972 (local address 127.0.98.62:34323)
0619 14:50:49.576266 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:49.576271 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:49.576423 (+ 152us) server_negotiation.cc:407] Connection header received
0619 14:50:49.576716 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:49.576725 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:49.576771 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:49.576879 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:49.577694 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:49.578271 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:49.579252 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:49.579507 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:49.579633 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:49.581469 (+ 1836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:49.581472 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:49.581481 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:49.581511 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:49.583125 (+ 1614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:49.583612 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:49.583616 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:49.583618 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:49.583706 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:49.584027 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:49.584031 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:49.584033 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:49.584286 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:49.584443 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:49.584744 (+ 301us) server_negotiation.cc:299] Negotiation successful
0619 14:50:49.584881 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179}
I20260619 14:50:49.589612 17434 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43972:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:50:49.593189 17434 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:50:49.610211 17904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:49.603054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41973 (local address 127.0.0.1:33506)
0619 14:50:49.603803 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:49.603860 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:49.604017 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:49.605043 (+ 1026us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:49.605052 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:49.605095 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:50:49.605275 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:49.605283 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:49.606670 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:49.606676 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:49.608304 (+ 1628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:49.608315 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:49.609570 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:49.609574 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:49.609695 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:49.609734 (+ 39us) client_negotiation.cc:769] Sending connection context
0619 14:50:49.609860 (+ 126us) client_negotiation.cc:240] Negotiation successful
0619 14:50:49.610021 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"mutex_wait_us":145,"spinlock_wait_cycles":14336,"thread_start_us":111,"threads_started":1}
I20260619 14:50:49.610407 17908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:49.604057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46116 (local address 127.0.98.2:40125)
0619 14:50:49.604496 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:49.604514 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:49.604537 (+ 23us) server_negotiation.cc:407] Connection header received
0619 14:50:49.604620 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:49.604625 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:49.604869 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:49.605055 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:50:49.605402 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:49.606716 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:49.608788 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:49.609645 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:49.609771 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:49.609894 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:49.610115 (+ 221us) server_negotiation.cc:299] Negotiation successful
0619 14:50:49.610247 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":121,"threads_started":1}
I20260619 14:50:49.610790 17906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:49.603309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33506 (local address 127.0.98.1:41973)
0619 14:50:49.604512 (+ 1203us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:49.604519 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:49.604540 (+ 21us) server_negotiation.cc:407] Connection header received
0619 14:50:49.604620 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:49.604628 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:49.604878 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:49.605425 (+ 547us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:50:49.605573 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:49.606557 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:49.608714 (+ 2157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:49.609424 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:49.609567 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:49.609759 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:49.610455 (+ 696us) server_negotiation.cc:299] Negotiation successful
0619 14:50:49.610644 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1068,"thread_start_us":97,"threads_started":1}
I20260619 14:50:49.610220 17905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:49.603298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40125 (local address 127.0.0.1:46116)
0619 14:50:49.604145 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:49.604158 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:49.604242 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:49.605050 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:49.605053 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:49.605086 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:50:49.605282 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:49.605289 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:49.606876 (+ 1587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:49.606879 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:49.608625 (+ 1746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:49.608632 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:49.609848 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:49.609855 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:49.609951 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:49.609972 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 14:50:49.610046 (+ 74us) client_negotiation.cc:240] Negotiation successful
0619 14:50:49.610104 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":788,"mutex_wait_us":189,"spinlock_wait_cycles":1280,"thread_start_us":91,"threads_started":1}
I20260619 14:50:49.611388 17909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:49.603299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37155 (local address 127.0.0.1:53264)
0619 14:50:49.604412 (+ 1113us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:49.604426 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:49.604583 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:49.605313 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:49.605319 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:49.605338 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:50:49.605432 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:49.605437 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:49.606903 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:49.606906 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:49.608615 (+ 1709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:49.608627 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:49.610005 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:49.610008 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:49.610139 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:49.610161 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 14:50:49.611171 (+ 1010us) client_negotiation.cc:240] Negotiation successful
0619 14:50:49.611266 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1054,"mutex_wait_us":1026,"thread_start_us":97,"threads_started":1}
I20260619 14:50:49.612066 17907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:49.603540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53264 (local address 127.0.98.3:37155)
0619 14:50:49.603985 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:49.603998 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:49.604512 (+ 514us) server_negotiation.cc:407] Connection header received
0619 14:50:49.604818 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:49.604823 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:49.605001 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:49.605120 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:50:49.605551 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:49.606759 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:49.608788 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:49.609642 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:49.609745 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:49.609891 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:49.610296 (+ 405us) server_negotiation.cc:299] Negotiation successful
0619 14:50:49.610860 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":142,"threads_started":1}
I20260619 14:50:49.614408 17559 tablet_service.cc:1467] Processing CreateTablet for tablet 2880ccab46dc485892d899a3fda1280b (DEFAULT_TABLE table=test-table [id=fb779f038d9d4b3199e6e2916c4aca46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:50:49.614687 17693 tablet_service.cc:1467] Processing CreateTablet for tablet 2880ccab46dc485892d899a3fda1280b (DEFAULT_TABLE table=test-table [id=fb779f038d9d4b3199e6e2916c4aca46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:50:49.614881 17827 tablet_service.cc:1467] Processing CreateTablet for tablet 2880ccab46dc485892d899a3fda1280b (DEFAULT_TABLE table=test-table [id=fb779f038d9d4b3199e6e2916c4aca46]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:50:49.615710 17693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2880ccab46dc485892d899a3fda1280b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:50:49.615710 17559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2880ccab46dc485892d899a3fda1280b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:50:49.616066 17827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2880ccab46dc485892d899a3fda1280b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:50:49.623878 17910 tablet_bootstrap.cc:492] T 2880ccab46dc485892d899a3fda1280b P 0b7e9761278c49b59bc8263c08839b3e: Bootstrap starting.
I20260619 14:50:49.624610 17912 tablet_bootstrap.cc:492] T 2880ccab46dc485892d899a3fda1280b P f2250f53fc534800987e77f3c8f5d0e7: Bootstrap starting.
I20260619 14:50:49.624876 17911 tablet_bootstrap.cc:492] T 2880ccab46dc485892d899a3fda1280b P 02fc9904da434210a6a7fda483348222: Bootstrap starting.
I20260619 14:50:49.627131 17911 tablet_bootstrap.cc:654] T 2880ccab46dc485892d899a3fda1280b P 02fc9904da434210a6a7fda483348222: Neither blocks nor log segments found. Creating new log.
I20260619 14:50:49.627466 17910 tablet_bootstrap.cc:654] T 2880ccab46dc485892d899a3fda1280b P 0b7e9761278c49b59bc8263c08839b3e: Neither blocks nor log segments found. Creating new log.
I20260619 14:50:49.627553 17912 tablet_bootstrap.cc:654] T 2880ccab46dc485892d899a3fda1280b P f2250f53fc534800987e77f3c8f5d0e7: Neither blocks nor log segments found. Creating new log.
I20260619 14:50:49.628162 17911 log.cc:824] T 2880ccab46dc485892d899a3fda1280b P 02fc9904da434210a6a7fda483348222: Log is configured to *not* fsync() on all Append() calls
I20260619 14:50:49.628306 17912 log.cc:824] T 2880ccab46dc485892d899a3fda1280b P f2250f53fc534800987e77f3c8f5d0e7: Log is configured to *not* fsync() on all Append() calls
I20260619 14:50:49.628683 17910 log.cc:824] T 2880ccab46dc485892d899a3fda1280b P 0b7e9761278c49b59bc8263c08839b3e: Log is configured to *not* fsync() on all Append() calls
I20260619 14:50:49.630371 17912 tablet_bootstrap.cc:492] T 2880ccab46dc485892d899a3fda1280b P f2250f53fc534800987e77f3c8f5d0e7: No bootstrap required, opened a new log
I20260619 14:50:49.630522 17910 tablet_bootstrap.cc:492] T 2880ccab46dc485892d899a3fda1280b P 0b7e9761278c49b59bc8263c08839b3e: No bootstrap required, opened a new log
I20260619 14:50:49.630527 17911 tablet_bootstrap.cc:492] T 2880ccab46dc485892d899a3fda1280b P 02fc9904da434210a6a7fda483348222: No bootstrap required, opened a new log
I20260619 14:50:49.630662 17912 ts_tablet_manager.cc:1397] T 2880ccab46dc485892d899a3fda1280b P f2250f53fc534800987e77f3c8f5d0e7: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 14:50:49.630724 17911 ts_tablet_manager.cc:1397] T 2880ccab46dc485892d899a3fda1280b P 02fc9904da434210a6a7fda483348222: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 14:50:49.630724 17910 ts_tablet_manager.cc:1397] T 2880ccab46dc485892d899a3fda1280b P 0b7e9761278c49b59bc8263c08839b3e: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260619 14:50:49.633931 17912 raft_consensus.cc:348] T 2880ccab46dc485892d899a3fda1280b P f2250f53fc534800987e77f3c8f5d0e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fc9904da434210a6a7fda483348222" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41973 } } peers { permanent_uuid: "0b7e9761278c49b59bc8263c08839b3e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37155 } } peers { permanent_uuid: "f2250f53fc534800987e77f3c8f5d0e7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40125 } }
I20260619 14:50:49.634258 17912 raft_consensus.cc:374] T 2880ccab46dc485892d899a3fda1280b P f2250f53fc534800987e77f3c8f5d0e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:50:49.634310 17912 raft_consensus.cc:729] T 2880ccab46dc485892d899a3fda1280b P f2250f53fc534800987e77f3c8f5d0e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2250f53fc534800987e77f3c8f5d0e7, State: Initialized, Role: FOLLOWER
I20260619 14:50:49.634871 17912 consensus_queue.cc:260] T 2880ccab46dc485892d899a3fda1280b P f2250f53fc534800987e77f3c8f5d0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fc9904da434210a6a7fda483348222" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41973 } } peers { permanent_uuid: "0b7e9761278c49b59bc8263c08839b3e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37155 } } peers { permanent_uuid: "f2250f53fc534800987e77f3c8f5d0e7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40125 } }
I20260619 14:50:49.635715 17759 heartbeater.cc:503] Master 127.0.98.62:34323 was elected leader, sending a full tablet report...
I20260619 14:50:49.635524 17911 raft_consensus.cc:348] T 2880ccab46dc485892d899a3fda1280b P 02fc9904da434210a6a7fda483348222 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fc9904da434210a6a7fda483348222" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41973 } } peers { permanent_uuid: "0b7e9761278c49b59bc8263c08839b3e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37155 } } peers { permanent_uuid: "f2250f53fc534800987e77f3c8f5d0e7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40125 } }
I20260619 14:50:49.635816 17911 raft_consensus.cc:374] T 2880ccab46dc485892d899a3fda1280b P 02fc9904da434210a6a7fda483348222 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:50:49.635877 17911 raft_consensus.cc:729] T 2880ccab46dc485892d899a3fda1280b P 02fc9904da434210a6a7fda483348222 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02fc9904da434210a6a7fda483348222, State: Initialized, Role: FOLLOWER
I20260619 14:50:49.635751 17910 raft_consensus.cc:348] T 2880ccab46dc485892d899a3fda1280b P 0b7e9761278c49b59bc8263c08839b3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fc9904da434210a6a7fda483348222" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41973 } } peers { permanent_uuid: "0b7e9761278c49b59bc8263c08839b3e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37155 } } peers { permanent_uuid: "f2250f53fc534800987e77f3c8f5d0e7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40125 } }
I20260619 14:50:49.636015 17910 raft_consensus.cc:374] T 2880ccab46dc485892d899a3fda1280b P 0b7e9761278c49b59bc8263c08839b3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:50:49.636068 17910 raft_consensus.cc:729] T 2880ccab46dc485892d899a3fda1280b P 0b7e9761278c49b59bc8263c08839b3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b7e9761278c49b59bc8263c08839b3e, State: Initialized, Role: FOLLOWER
I20260619 14:50:49.636369 17911 consensus_queue.cc:260] T 2880ccab46dc485892d899a3fda1280b P 02fc9904da434210a6a7fda483348222 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fc9904da434210a6a7fda483348222" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41973 } } peers { permanent_uuid: "0b7e9761278c49b59bc8263c08839b3e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37155 } } peers { permanent_uuid: "f2250f53fc534800987e77f3c8f5d0e7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40125 } }
I20260619 14:50:49.636408 17910 consensus_queue.cc:260] T 2880ccab46dc485892d899a3fda1280b P 0b7e9761278c49b59bc8263c08839b3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fc9904da434210a6a7fda483348222" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41973 } } peers { permanent_uuid: "0b7e9761278c49b59bc8263c08839b3e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37155 } } peers { permanent_uuid: "f2250f53fc534800987e77f3c8f5d0e7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40125 } }
I20260619 14:50:49.637435 17910 ts_tablet_manager.cc:1428] T 2880ccab46dc485892d899a3fda1280b P 0b7e9761278c49b59bc8263c08839b3e: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260619 14:50:49.637436 17911 ts_tablet_manager.cc:1428] T 2880ccab46dc485892d899a3fda1280b P 02fc9904da434210a6a7fda483348222: Time spent starting tablet: real 0.007s user 0.004s sys 0.003s
I20260619 14:50:49.637440 17912 ts_tablet_manager.cc:1428] T 2880ccab46dc485892d899a3fda1280b P f2250f53fc534800987e77f3c8f5d0e7: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260619 14:50:49.637805 17893 heartbeater.cc:503] Master 127.0.98.62:34323 was elected leader, sending a full tablet report...
I20260619 14:50:49.637806 17625 heartbeater.cc:503] Master 127.0.98.62:34323 was elected leader, sending a full tablet report...
W20260619 14:50:49.776898 17626 tablet.cc:2369] T 2880ccab46dc485892d899a3fda1280b P 02fc9904da434210a6a7fda483348222: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 14:50:49.780001 17760 tablet.cc:2369] T 2880ccab46dc485892d899a3fda1280b P f2250f53fc534800987e77f3c8f5d0e7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 14:50:49.791313 17894 tablet.cc:2369] T 2880ccab46dc485892d899a3fda1280b P 0b7e9761278c49b59bc8263c08839b3e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:50:49.812489 17918 raft_consensus.cc:482] T 2880ccab46dc485892d899a3fda1280b P 0b7e9761278c49b59bc8263c08839b3e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:50:49.812817 17918 raft_consensus.cc:504] T 2880ccab46dc485892d899a3fda1280b P 0b7e9761278c49b59bc8263c08839b3e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fc9904da434210a6a7fda483348222" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41973 } } peers { permanent_uuid: "0b7e9761278c49b59bc8263c08839b3e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37155 } } peers { permanent_uuid: "f2250f53fc534800987e77f3c8f5d0e7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40125 } }
I20260619 14:50:49.814553 17918 leader_election.cc:288] T 2880ccab46dc485892d899a3fda1280b P 0b7e9761278c49b59bc8263c08839b3e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 02fc9904da434210a6a7fda483348222 (127.0.98.1:41973), f2250f53fc534800987e77f3c8f5d0e7 (127.0.98.2:40125)
I20260619 14:50:49.819383 17895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:49.814783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41973 (local address 127.0.98.3:57987)
0619 14:50:49.815022 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:49.815042 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:49.815158 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:49.815597 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:49.815600 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:49.815622 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:50:49.815763 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:49.815779 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:49.816973 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:49.816977 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:49.818042 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:49.818054 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:49.818904 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:49.818909 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:49.819039 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:49.819073 (+ 34us) client_negotiation.cc:769] Sending connection context
0619 14:50:49.819155 (+ 82us) client_negotiation.cc:240] Negotiation successful
0619 14:50:49.819251 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":3}
I20260619 14:50:49.819491 17906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:49.814948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57987 (local address 127.0.98.1:41973)
0619 14:50:49.815076 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:49.815081 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:49.815099 (+ 18us) server_negotiation.cc:407] Connection header received
0619 14:50:49.815287 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:49.815290 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:49.815367 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:49.815455 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:50:49.816032 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:49.816849 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:49.818212 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:49.818769 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:49.818860 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:49.818914 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:49.819234 (+ 320us) server_negotiation.cc:299] Negotiation successful
0619 14:50:49.819328 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 14:50:49.819651 17919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:49.814933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40125 (local address 127.0.98.3:38273)
0619 14:50:49.815288 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:49.815301 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:49.815428 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:49.815803 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:49.815806 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:49.815967 (+ 161us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:50:49.816106 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:49.816114 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:49.817219 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:49.817221 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:49.818231 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:49.818242 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:49.819208 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:49.819212 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:49.819310 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:49.819332 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 14:50:49.819431 (+ 99us) client_negotiation.cc:240] Negotiation successful
0619 14:50:49.819496 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"mutex_wait_us":70,"thread_start_us":162,"threads_started":1}
I20260619 14:50:49.819710 17908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:49.815091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38273 (local address 127.0.98.2:40125)
0619 14:50:49.815203 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:49.815207 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:49.815547 (+ 340us) server_negotiation.cc:407] Connection header received
0619 14:50:49.815608 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:49.815611 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:49.815659 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:49.815762 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:50:49.816277 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:49.817087 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:49.818396 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:49.818954 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:49.819091 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:49.819179 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:49.819474 (+ 295us) server_negotiation.cc:299] Negotiation successful
0619 14:50:49.819565 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 14:50:49.820259 17579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2880ccab46dc485892d899a3fda1280b" candidate_uuid: "0b7e9761278c49b59bc8263c08839b3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02fc9904da434210a6a7fda483348222" is_pre_election: true
I20260619 14:50:49.820451 17713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2880ccab46dc485892d899a3fda1280b" candidate_uuid: "0b7e9761278c49b59bc8263c08839b3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2250f53fc534800987e77f3c8f5d0e7" is_pre_election: true
I20260619 14:50:49.820755 17713 raft_consensus.cc:2454] T 2880ccab46dc485892d899a3fda1280b P f2250f53fc534800987e77f3c8f5d0e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b7e9761278c49b59bc8263c08839b3e in term 0.
I20260619 14:50:49.820794 17579 raft_consensus.cc:2454] T 2880ccab46dc485892d899a3fda1280b P 02fc9904da434210a6a7fda483348222 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b7e9761278c49b59bc8263c08839b3e in term 0.
I20260619 14:50:49.821367 17781 leader_election.cc:302] T 2880ccab46dc485892d899a3fda1280b P 0b7e9761278c49b59bc8263c08839b3e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02fc9904da434210a6a7fda483348222, 0b7e9761278c49b59bc8263c08839b3e; no voters:
I20260619 14:50:49.821741 17918 raft_consensus.cc:2781] T 2880ccab46dc485892d899a3fda1280b P 0b7e9761278c49b59bc8263c08839b3e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:50:49.821830 17918 raft_consensus.cc:482] T 2880ccab46dc485892d899a3fda1280b P 0b7e9761278c49b59bc8263c08839b3e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:50:49.821871 17918 raft_consensus.cc:3037] T 2880ccab46dc485892d899a3fda1280b P 0b7e9761278c49b59bc8263c08839b3e [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:50:49.822995 17918 raft_consensus.cc:504] T 2880ccab46dc485892d899a3fda1280b P 0b7e9761278c49b59bc8263c08839b3e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fc9904da434210a6a7fda483348222" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41973 } } peers { permanent_uuid: "0b7e9761278c49b59bc8263c08839b3e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37155 } } peers { permanent_uuid: "f2250f53fc534800987e77f3c8f5d0e7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40125 } }
I20260619 14:50:49.823387 17918 leader_election.cc:288] T 2880ccab46dc485892d899a3fda1280b P 0b7e9761278c49b59bc8263c08839b3e [CANDIDATE]: Term 1 election: Requested vote from peers 02fc9904da434210a6a7fda483348222 (127.0.98.1:41973), f2250f53fc534800987e77f3c8f5d0e7 (127.0.98.2:40125)
I20260619 14:50:49.823823 17713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2880ccab46dc485892d899a3fda1280b" candidate_uuid: "0b7e9761278c49b59bc8263c08839b3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2250f53fc534800987e77f3c8f5d0e7"
I20260619 14:50:49.823872 17579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2880ccab46dc485892d899a3fda1280b" candidate_uuid: "0b7e9761278c49b59bc8263c08839b3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02fc9904da434210a6a7fda483348222"
I20260619 14:50:49.823974 17713 raft_consensus.cc:3037] T 2880ccab46dc485892d899a3fda1280b P f2250f53fc534800987e77f3c8f5d0e7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:50:49.824038 17579 raft_consensus.cc:3037] T 2880ccab46dc485892d899a3fda1280b P 02fc9904da434210a6a7fda483348222 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:50:49.825166 17713 raft_consensus.cc:2454] T 2880ccab46dc485892d899a3fda1280b P f2250f53fc534800987e77f3c8f5d0e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b7e9761278c49b59bc8263c08839b3e in term 1.
I20260619 14:50:49.825511 17579 raft_consensus.cc:2454] T 2880ccab46dc485892d899a3fda1280b P 02fc9904da434210a6a7fda483348222 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b7e9761278c49b59bc8263c08839b3e in term 1.
I20260619 14:50:49.825564 17782 leader_election.cc:302] T 2880ccab46dc485892d899a3fda1280b P 0b7e9761278c49b59bc8263c08839b3e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b7e9761278c49b59bc8263c08839b3e, f2250f53fc534800987e77f3c8f5d0e7; no voters:
I20260619 14:50:49.825742 17918 raft_consensus.cc:2781] T 2880ccab46dc485892d899a3fda1280b P 0b7e9761278c49b59bc8263c08839b3e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:50:49.825995 17918 raft_consensus.cc:686] T 2880ccab46dc485892d899a3fda1280b P 0b7e9761278c49b59bc8263c08839b3e [term 1 LEADER]: Becoming Leader. State: Replica: 0b7e9761278c49b59bc8263c08839b3e, State: Running, Role: LEADER
I20260619 14:50:49.826372 17918 consensus_queue.cc:237] T 2880ccab46dc485892d899a3fda1280b P 0b7e9761278c49b59bc8263c08839b3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fc9904da434210a6a7fda483348222" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41973 } } peers { permanent_uuid: "0b7e9761278c49b59bc8263c08839b3e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37155 } } peers { permanent_uuid: "f2250f53fc534800987e77f3c8f5d0e7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40125 } }
I20260619 14:50:49.829771 17434 catalog_manager.cc:5495] T 2880ccab46dc485892d899a3fda1280b P 0b7e9761278c49b59bc8263c08839b3e reported cstate change: term changed from 0 to 1, leader changed from <none> to 0b7e9761278c49b59bc8263c08839b3e (127.0.98.3). New cstate: current_term: 1 leader_uuid: "0b7e9761278c49b59bc8263c08839b3e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02fc9904da434210a6a7fda483348222" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b7e9761278c49b59bc8263c08839b3e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f2250f53fc534800987e77f3c8f5d0e7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40125 } health_report { overall_health: UNKNOWN } } }
I20260619 14:50:49.851868 17907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:49.847037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53280 (local address 127.0.98.3:37155)
0619 14:50:49.847190 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:49.847193 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:49.847210 (+ 17us) server_negotiation.cc:407] Connection header received
0619 14:50:49.847414 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:49.847417 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:49.847481 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:49.847583 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:50:49.848014 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:49.848838 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:49.849883 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:49.850131 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:49.850223 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:49.850866 (+ 643us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:50:49.851332 (+ 466us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:50:49.851579 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:49.851611 (+ 32us) server_negotiation.cc:299] Negotiation successful
0619 14:50:49.851668 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260619 14:50:49.858881 17579 raft_consensus.cc:1261] T 2880ccab46dc485892d899a3fda1280b P 02fc9904da434210a6a7fda483348222 [term 1 FOLLOWER]: Refusing update from remote peer 0b7e9761278c49b59bc8263c08839b3e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:50:49.858919 17713 raft_consensus.cc:1261] T 2880ccab46dc485892d899a3fda1280b P f2250f53fc534800987e77f3c8f5d0e7 [term 1 FOLLOWER]: Refusing update from remote peer 0b7e9761278c49b59bc8263c08839b3e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:50:49.859721 17920 consensus_queue.cc:1035] T 2880ccab46dc485892d899a3fda1280b P 0b7e9761278c49b59bc8263c08839b3e [LEADER]: Connected to new peer: Peer: permanent_uuid: "02fc9904da434210a6a7fda483348222" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:50:49.859978 17918 consensus_queue.cc:1035] T 2880ccab46dc485892d899a3fda1280b P 0b7e9761278c49b59bc8263c08839b3e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2250f53fc534800987e77f3c8f5d0e7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:50:49.870834 17924 mvcc.cc:205] Tried to move back new op lower bound from 7298583141812658176 to 7298583141691879424. Current Snapshot: MvccSnapshot[applied={T|T < 7298583141812658176}]
I20260619 14:50:49.872210 17925 mvcc.cc:205] Tried to move back new op lower bound from 7298583141812658176 to 7298583141691879424. Current Snapshot: MvccSnapshot[applied={T|T < 7298583141812658176}]
I20260619 14:50:49.880704 17433 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43972:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:50:49.880944 17433 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43972:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:50:49.883951 17433 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7bb2489393334cfbaad62db8963ca9f0: Sending DeleteTablet for 3 replicas of tablet 2880ccab46dc485892d899a3fda1280b
I20260619 14:50:49.885102 17559 tablet_service.cc:1514] Processing DeleteTablet for tablet 2880ccab46dc485892d899a3fda1280b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:50:49 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33506
I20260619 14:50:49.885161 17827 tablet_service.cc:1514] Processing DeleteTablet for tablet 2880ccab46dc485892d899a3fda1280b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:50:49 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53264
I20260619 14:50:49.885159 17693 tablet_service.cc:1514] Processing DeleteTablet for tablet 2880ccab46dc485892d899a3fda1280b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:50:49 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46116
I20260619 14:50:49.885742 17934 tablet_replica.cc:330] T 2880ccab46dc485892d899a3fda1280b P 02fc9904da434210a6a7fda483348222: stopping tablet replica
I20260619 14:50:49.885931 17935 tablet_replica.cc:330] T 2880ccab46dc485892d899a3fda1280b P 0b7e9761278c49b59bc8263c08839b3e: stopping tablet replica
I20260619 14:50:49.886135 17934 raft_consensus.cc:2229] T 2880ccab46dc485892d899a3fda1280b P 02fc9904da434210a6a7fda483348222 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:50:49.886327 17935 raft_consensus.cc:2229] T 2880ccab46dc485892d899a3fda1280b P 0b7e9761278c49b59bc8263c08839b3e [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:50:49.886551 17934 raft_consensus.cc:2258] T 2880ccab46dc485892d899a3fda1280b P 02fc9904da434210a6a7fda483348222 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:50:49.886619 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 17496
I20260619 14:50:49.888016 17935 raft_consensus.cc:2258] T 2880ccab46dc485892d899a3fda1280b P 0b7e9761278c49b59bc8263c08839b3e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:50:49.889932 17933 tablet_replica.cc:330] T 2880ccab46dc485892d899a3fda1280b P f2250f53fc534800987e77f3c8f5d0e7: stopping tablet replica
I20260619 14:50:49.890331 17933 raft_consensus.cc:2229] T 2880ccab46dc485892d899a3fda1280b P f2250f53fc534800987e77f3c8f5d0e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:50:49.890707 17933 raft_consensus.cc:2258] T 2880ccab46dc485892d899a3fda1280b P f2250f53fc534800987e77f3c8f5d0e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:50:49.892622 17933 ts_tablet_manager.cc:1905] T 2880ccab46dc485892d899a3fda1280b P f2250f53fc534800987e77f3c8f5d0e7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:50:49.893945 17935 ts_tablet_manager.cc:1905] T 2880ccab46dc485892d899a3fda1280b P 0b7e9761278c49b59bc8263c08839b3e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:50:49.894054 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 17630
W20260619 14:50:49.894057 17419 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41973 (error 108)
I20260619 14:50:49.896921 17905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:49.896527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41973 (local address 127.0.0.1:33520)
0619 14:50:49.896715 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:49.896821 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260619 14:50:49.897161 17419 catalog_manager.cc:4584] TS 02fc9904da434210a6a7fda483348222 (127.0.98.1:41973): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2880ccab46dc485892d899a3fda1280b: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41973: connect: Connection refused (error 111)
I20260619 14:50:49.898202 17935 ts_tablet_manager.cc:1918] T 2880ccab46dc485892d899a3fda1280b P 0b7e9761278c49b59bc8263c08839b3e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:50:49.898380 17935 log.cc:1196] T 2880ccab46dc485892d899a3fda1280b P 0b7e9761278c49b59bc8263c08839b3e: Deleting WAL directory at /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781880383132139-392-0/minicluster-data/ts-2/wal/wals/2880ccab46dc485892d899a3fda1280b
I20260619 14:50:49.898738 17935 ts_tablet_manager.cc:1939] T 2880ccab46dc485892d899a3fda1280b P 0b7e9761278c49b59bc8263c08839b3e: Deleting consensus metadata
I20260619 14:50:49.899995 17418 catalog_manager.cc:4841] TS 0b7e9761278c49b59bc8263c08839b3e (127.0.98.3:37155): tablet 2880ccab46dc485892d899a3fda1280b (table test-table [id=fb779f038d9d4b3199e6e2916c4aca46]) successfully deleted
I20260619 14:50:49.901378 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 17764
I20260619 14:50:49.903147 17905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:49.902819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40125 (local address 127.0.0.1:46124)
0619 14:50:49.902989 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:49.903055 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:40125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260619 14:50:49.907653 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 17402
2026-06-19T14:50:49Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2696 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-19T14:50:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:50:49Z Disabled control of system clock
I20260619 14:50:49.942083 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880383132139-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880383132139-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880383132139-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46749
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:32857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46749
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 14:50:50.071025 17940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:50:50.071365 17940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:50:50.071408 17940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:50:50.076087 17940 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:50:50.076184 17940 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:50:50.076202 17940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:50:50.076222 17940 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:50:50.076239 17940 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:50:50.082372 17940 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880383132139-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880383132139-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46749
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46749
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17940
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880383132139-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:50:50.083612 17940 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:50:50.085889 17940 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:50:50.096011 17948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:50:50.096123 17940 server_base.cc:1034] running on GCE node
W20260619 14:50:50.096071 17945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:50.096071 17946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:50:50.097188 17940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:50:50.098495 17940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:50:50.099730 17940 hybrid_clock.cc:648] HybridClock initialized: now 1781880650099699 us; error 49 us; skew 500 ppm
I20260619 14:50:50.102300 17940 webserver.cc:460] Webserver started at http://127.0.98.62:43461/ using document root <none> and password file <none>
I20260619 14:50:50.103154 17940 fs_manager.cc:359] Metadata directory not provided
I20260619 14:50:50.103293 17940 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:50:50.103562 17940 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:50:50.105888 17940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880383132139-392-0/minicluster-data/master-0/data/instance:
uuid: "66644343d480449d9ae2168c2dc7f97c"
format_stamp: "Formatted at 2026-06-19 14:50:50 on dist-test-slave-2sb1"
server_key: "9064a1ff5298da7c9e55da05a250a0d1"
server_key_iv: "1aff705aff556c93a151d29563d1d614"
server_key_version: "encryptionkey@0"
I20260619 14:50:50.106540 17940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880383132139-392-0/minicluster-data/master-0/wal/instance:
uuid: "66644343d480449d9ae2168c2dc7f97c"
format_stamp: "Formatted at 2026-06-19 14:50:50 on dist-test-slave-2sb1"
server_key: "9064a1ff5298da7c9e55da05a250a0d1"
server_key_iv: "1aff705aff556c93a151d29563d1d614"
server_key_version: "encryptionkey@0"
I20260619 14:50:50.111953 17940 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260619 14:50:50.115456 17954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:50.117462 17940 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260619 14:50:50.117684 17940 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880383132139-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880383132139-392-0/minicluster-data/master-0/wal
uuid: "66644343d480449d9ae2168c2dc7f97c"
format_stamp: "Formatted at 2026-06-19 14:50:50 on dist-test-slave-2sb1"
server_key: "9064a1ff5298da7c9e55da05a250a0d1"
server_key_iv: "1aff705aff556c93a151d29563d1d614"
server_key_version: "encryptionkey@0"
I20260619 14:50:50.117844 17940 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880383132139-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880383132139-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880383132139-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:50:50.148052 17940 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:50:50.148978 17940 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:50:50.149199 17940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:50:50.158486 17940 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46749
I20260619 14:50:50.158545 18006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46749 every 8 connection(s)
I20260619 14:50:50.159673 17940 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:50:50.160876 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 17940
I20260619 14:50:50.161082 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880383132139-392-0/minicluster-data/master-0/wal/instance
I20260619 14:50:50.161377 392 external_mini_cluster.cc:1428] Setting key ba4e8bd578b2f056b47ff02f887a8afb
I20260619 14:50:50.163661 18007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:50:50.171886 18007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66644343d480449d9ae2168c2dc7f97c: Bootstrap starting.
I20260619 14:50:50.173710 18010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:50.162825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36180 (local address 127.0.98.62:46749)
0619 14:50:50.163585 (+ 760us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:50.163608 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:50.163656 (+ 48us) server_negotiation.cc:407] Connection header received
0619 14:50:50.164619 (+ 963us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:50.164672 (+ 53us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:50.165221 (+ 549us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:50.165653 (+ 432us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:50.166143 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:50.167846 (+ 1703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:50.169023 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:50.169817 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:50.170025 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:50.171285 (+ 1260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:50.171293 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:50.171312 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:50:50.171878 (+ 566us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:50.171962 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:50:50.171997 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:50.172108 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:50.172374 (+ 266us) server_negotiation.cc:299] Negotiation successful
0619 14:50:50.172866 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":145,"threads_started":1}
I20260619 14:50:50.174654 18007 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 66644343d480449d9ae2168c2dc7f97c: Neither blocks nor log segments found. Creating new log.
I20260619 14:50:50.175513 18007 log.cc:824] T 00000000000000000000000000000000 P 66644343d480449d9ae2168c2dc7f97c: Log is configured to *not* fsync() on all Append() calls
I20260619 14:50:50.177807 18007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66644343d480449d9ae2168c2dc7f97c: No bootstrap required, opened a new log
I20260619 14:50:50.181947 18007 raft_consensus.cc:348] T 00000000000000000000000000000000 P 66644343d480449d9ae2168c2dc7f97c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66644343d480449d9ae2168c2dc7f97c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46749 } }
I20260619 14:50:50.182262 18007 raft_consensus.cc:374] T 00000000000000000000000000000000 P 66644343d480449d9ae2168c2dc7f97c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:50:50.182364 18007 raft_consensus.cc:729] T 00000000000000000000000000000000 P 66644343d480449d9ae2168c2dc7f97c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66644343d480449d9ae2168c2dc7f97c, State: Initialized, Role: FOLLOWER
I20260619 14:50:50.182924 18007 consensus_queue.cc:260] T 00000000000000000000000000000000 P 66644343d480449d9ae2168c2dc7f97c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66644343d480449d9ae2168c2dc7f97c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46749 } }
I20260619 14:50:50.183105 18007 raft_consensus.cc:388] T 00000000000000000000000000000000 P 66644343d480449d9ae2168c2dc7f97c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:50:50.183203 18007 raft_consensus.cc:482] T 00000000000000000000000000000000 P 66644343d480449d9ae2168c2dc7f97c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:50:50.183319 18007 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66644343d480449d9ae2168c2dc7f97c [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:50:50.184403 18007 raft_consensus.cc:504] T 00000000000000000000000000000000 P 66644343d480449d9ae2168c2dc7f97c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66644343d480449d9ae2168c2dc7f97c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46749 } }
I20260619 14:50:50.184862 18007 leader_election.cc:302] T 00000000000000000000000000000000 P 66644343d480449d9ae2168c2dc7f97c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 66644343d480449d9ae2168c2dc7f97c; no voters:
I20260619 14:50:50.185271 18007 leader_election.cc:288] T 00000000000000000000000000000000 P 66644343d480449d9ae2168c2dc7f97c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:50:50.185463 18012 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 66644343d480449d9ae2168c2dc7f97c [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:50:50.185760 18012 raft_consensus.cc:686] T 00000000000000000000000000000000 P 66644343d480449d9ae2168c2dc7f97c [term 1 LEADER]: Becoming Leader. State: Replica: 66644343d480449d9ae2168c2dc7f97c, State: Running, Role: LEADER
I20260619 14:50:50.186223 18012 consensus_queue.cc:237] T 00000000000000000000000000000000 P 66644343d480449d9ae2168c2dc7f97c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66644343d480449d9ae2168c2dc7f97c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46749 } }
I20260619 14:50:50.186748 18007 sys_catalog.cc:564] T 00000000000000000000000000000000 P 66644343d480449d9ae2168c2dc7f97c [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:50:50.188778 18013 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66644343d480449d9ae2168c2dc7f97c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "66644343d480449d9ae2168c2dc7f97c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66644343d480449d9ae2168c2dc7f97c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46749 } } }
I20260619 14:50:50.188978 18013 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66644343d480449d9ae2168c2dc7f97c [sys.catalog]: This master's current role is: LEADER
I20260619 14:50:50.189270 18014 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66644343d480449d9ae2168c2dc7f97c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66644343d480449d9ae2168c2dc7f97c. Latest consensus state: current_term: 1 leader_uuid: "66644343d480449d9ae2168c2dc7f97c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66644343d480449d9ae2168c2dc7f97c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46749 } } }
I20260619 14:50:50.189388 18014 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66644343d480449d9ae2168c2dc7f97c [sys.catalog]: This master's current role is: LEADER
I20260619 14:50:50.189535 18021 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:50:50.193400 18021 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:50:50.201010 18021 catalog_manager.cc:1347] Generated new cluster ID: 0fad4021acd54591b0c2d264012ea94b
I20260619 14:50:50.201112 18021 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:50:50.213357 18021 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:50:50.214604 18021 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:50:50.221088 18021 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 66644343d480449d9ae2168c2dc7f97c: Generated new TSK 0
I20260619 14:50:50.222168 18021 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:50:50.227823 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880383132139-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880383132139-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880383132139-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880383132139-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46749
--builtin_ntp_servers=127.0.98.20:32857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 14:50:50.354972 18031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:50:50.355296 18031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:50:50.355340 18031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:50:50.359870 18031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:50:50.359968 18031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:50:50.360028 18031 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 14:50:50.365284 18031 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880383132139-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880383132139-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880383132139-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46749
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18031
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880383132139-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:50:50.366382 18031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:50:50.367805 18031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:50:50.378993 18039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:50.378993 18037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:50.379006 18036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:50:50.379406 18031 server_base.cc:1034] running on GCE node
I20260619 14:50:50.379886 18031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:50:50.380712 18031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:50:50.381999 18031 hybrid_clock.cc:648] HybridClock initialized: now 1781880650381964 us; error 55 us; skew 500 ppm
I20260619 14:50:50.384351 18031 webserver.cc:460] Webserver started at http://127.0.98.1:44015/ using document root <none> and password file <none>
I20260619 14:50:50.385160 18031 fs_manager.cc:359] Metadata directory not provided
I20260619 14:50:50.385272 18031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:50:50.385651 18031 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:50:50.388275 18031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880383132139-392-0/minicluster-data/ts-0/data/instance:
uuid: "4d18235f6f19499abee19b697dbbc993"
format_stamp: "Formatted at 2026-06-19 14:50:50 on dist-test-slave-2sb1"
server_key: "d1bbba5f7a426bea5d25e03a59584956"
server_key_iv: "fd073cf0580c102fd57a90faa749a20a"
server_key_version: "encryptionkey@0"
I20260619 14:50:50.389015 18031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880383132139-392-0/minicluster-data/ts-0/wal/instance:
uuid: "4d18235f6f19499abee19b697dbbc993"
format_stamp: "Formatted at 2026-06-19 14:50:50 on dist-test-slave-2sb1"
server_key: "d1bbba5f7a426bea5d25e03a59584956"
server_key_iv: "fd073cf0580c102fd57a90faa749a20a"
server_key_version: "encryptionkey@0"
I20260619 14:50:50.393265 18031 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 14:50:50.396229 18045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:50.397992 18031 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:50:50.398190 18031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880383132139-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880383132139-392-0/minicluster-data/ts-0/wal
uuid: "4d18235f6f19499abee19b697dbbc993"
format_stamp: "Formatted at 2026-06-19 14:50:50 on dist-test-slave-2sb1"
server_key: "d1bbba5f7a426bea5d25e03a59584956"
server_key_iv: "fd073cf0580c102fd57a90faa749a20a"
server_key_version: "encryptionkey@0"
I20260619 14:50:50.398352 18031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880383132139-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880383132139-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880383132139-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:50:50.430917 18031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:50:50.431838 18031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:50:50.432063 18031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:50:50.432830 18031 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:50:50.433879 18031 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:50:50.433934 18031 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:50.433979 18031 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:50:50.434021 18031 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:50.444353 18031 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33973
I20260619 14:50:50.444383 18158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33973 every 8 connection(s)
I20260619 14:50:50.445616 18031 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880383132139-392-0/minicluster-data/ts-0/data/info.pb
I20260619 14:50:50.445990 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 18031
I20260619 14:50:50.446120 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880383132139-392-0/minicluster-data/ts-0/wal/instance
I20260619 14:50:50.446563 392 external_mini_cluster.cc:1428] Setting key fb919075506841c0770fca107372637c
I20260619 14:50:50.448872 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880383132139-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880383132139-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880383132139-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880383132139-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46749
--builtin_ntp_servers=127.0.98.20:32857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 14:50:50.458544 18010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:50.447788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:32769 (local address 127.0.98.62:46749)
0619 14:50:50.448027 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:50.448032 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:50.449361 (+ 1329us) server_negotiation.cc:407] Connection header received
0619 14:50:50.451253 (+ 1892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:50.451258 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:50.451324 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:50.451424 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:50.452816 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:50.453530 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:50.455772 (+ 2242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:50.456023 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:50.456128 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:50.457811 (+ 1683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:50.457814 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:50.457816 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:50:50.457831 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:50.457853 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:50:50.457861 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:50.457932 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:50.458199 (+ 267us) server_negotiation.cc:299] Negotiation successful
0619 14:50:50.458273 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 14:50:50.459878 18161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:50.448349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46749 (local address 127.0.98.1:32769)
0619 14:50:50.449171 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:50.449238 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:50.451013 (+ 1775us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:50.451761 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:50.451773 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:50.451891 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:50.452615 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:50.452638 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:50.453689 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:50.453696 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:50.455403 (+ 1707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:50.455420 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:50.456235 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:50.456243 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:50.456531 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:50.457396 (+ 865us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:50:50.457400 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:50.457470 (+ 70us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:50:50.457580 (+ 110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:50:50.457962 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:50:50.457975 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:50:50.457985 (+ 10us) client_negotiation.cc:769] Sending connection context
0619 14:50:50.458141 (+ 156us) client_negotiation.cc:240] Negotiation successful
0619 14:50:50.458559 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":122,"threads_started":1}
I20260619 14:50:50.461792 18159 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46749
I20260619 14:50:50.462121 18159 heartbeater.cc:464] Registering TS with master...
I20260619 14:50:50.462703 18159 heartbeater.cc:511] Master 127.0.98.62:46749 requested a full tablet report, sending...
I20260619 14:50:50.464458 17971 ts_manager.cc:194] Registered new tserver with Master: 4d18235f6f19499abee19b697dbbc993 (127.0.98.1:33973)
I20260619 14:50:50.466136 17971 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:32769
W20260619 14:50:50.572958 18162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:50:50.573679 18162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:50:50.573731 18162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:50:50.578110 18162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:50:50.578179 18162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:50:50.578240 18162 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 14:50:50.583321 18162 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880383132139-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880383132139-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880383132139-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46749
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18162
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880383132139-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:50:50.584440 18162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:50:50.585986 18162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:50:50.596983 18167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:50.597044 18170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:50.597036 18168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:50:50.597321 18162 server_base.cc:1034] running on GCE node
I20260619 14:50:50.597791 18162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:50:50.598562 18162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:50:50.599812 18162 hybrid_clock.cc:648] HybridClock initialized: now 1781880650599760 us; error 66 us; skew 500 ppm
I20260619 14:50:50.602131 18162 webserver.cc:460] Webserver started at http://127.0.98.2:40143/ using document root <none> and password file <none>
I20260619 14:50:50.602820 18162 fs_manager.cc:359] Metadata directory not provided
I20260619 14:50:50.602945 18162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:50:50.603191 18162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:50:50.605523 18162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880383132139-392-0/minicluster-data/ts-1/data/instance:
uuid: "66c83651852747809a19d5295165b178"
format_stamp: "Formatted at 2026-06-19 14:50:50 on dist-test-slave-2sb1"
server_key: "426308a5c92ed7a444be6bbf17c11ee8"
server_key_iv: "cca101f2a7870e7cf3c190156eaf0178"
server_key_version: "encryptionkey@0"
I20260619 14:50:50.606184 18162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880383132139-392-0/minicluster-data/ts-1/wal/instance:
uuid: "66c83651852747809a19d5295165b178"
format_stamp: "Formatted at 2026-06-19 14:50:50 on dist-test-slave-2sb1"
server_key: "426308a5c92ed7a444be6bbf17c11ee8"
server_key_iv: "cca101f2a7870e7cf3c190156eaf0178"
server_key_version: "encryptionkey@0"
I20260619 14:50:50.610435 18162 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 14:50:50.614096 18176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:50.615957 18162 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260619 14:50:50.616178 18162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880383132139-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880383132139-392-0/minicluster-data/ts-1/wal
uuid: "66c83651852747809a19d5295165b178"
format_stamp: "Formatted at 2026-06-19 14:50:50 on dist-test-slave-2sb1"
server_key: "426308a5c92ed7a444be6bbf17c11ee8"
server_key_iv: "cca101f2a7870e7cf3c190156eaf0178"
server_key_version: "encryptionkey@0"
I20260619 14:50:50.616333 18162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880383132139-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880383132139-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880383132139-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:50:50.640412 18162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:50:50.641294 18162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:50:50.641530 18162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:50:50.642176 18162 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:50:50.643224 18162 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:50:50.643303 18162 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:50.643380 18162 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:50:50.643425 18162 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:50.655431 18162 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39693
I20260619 14:50:50.655488 18289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39693 every 8 connection(s)
I20260619 14:50:50.656447 18162 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880383132139-392-0/minicluster-data/ts-1/data/info.pb
I20260619 14:50:50.657303 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 18162
I20260619 14:50:50.657537 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880383132139-392-0/minicluster-data/ts-1/wal/instance
I20260619 14:50:50.657946 392 external_mini_cluster.cc:1428] Setting key 6849228fe304fd8e6e9441953deb34c2
I20260619 14:50:50.660768 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880383132139-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880383132139-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880383132139-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880383132139-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46749
--builtin_ntp_servers=127.0.98.20:32857
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 14:50:50.667629 18010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:50.658846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48609 (local address 127.0.98.62:46749)
0619 14:50:50.659069 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:50.659074 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:50.660597 (+ 1523us) server_negotiation.cc:407] Connection header received
0619 14:50:50.662329 (+ 1732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:50.662333 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:50.662391 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:50.662481 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:50.663600 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:50.664126 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:50.665427 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:50.665661 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:50.665779 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:50.667086 (+ 1307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:50.667089 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:50.667091 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:50:50.667106 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:50.667126 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:50:50.667134 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:50.667218 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:50.667402 (+ 184us) server_negotiation.cc:299] Negotiation successful
0619 14:50:50.667472 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
I20260619 14:50:50.668982 18292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:50.659591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46749 (local address 127.0.98.2:48609)
0619 14:50:50.660428 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:50.660474 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:50.661951 (+ 1477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:50.662701 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:50.662713 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:50.662826 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:50.663413 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:50.663428 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:50.664263 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:50.664266 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:50.665294 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:50.665304 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:50.665828 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:50.665832 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:50.666007 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:50.666739 (+ 732us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:50:50.666741 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:50.666790 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:50:50.666865 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:50:50.667221 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:50:50.667233 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:50:50.667242 (+ 9us) client_negotiation.cc:769] Sending connection context
0619 14:50:50.667402 (+ 160us) client_negotiation.cc:240] Negotiation successful
0619 14:50:50.667741 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":164,"threads_started":1}
I20260619 14:50:50.670559 18290 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46749
I20260619 14:50:50.670874 18290 heartbeater.cc:464] Registering TS with master...
I20260619 14:50:50.671702 18290 heartbeater.cc:511] Master 127.0.98.62:46749 requested a full tablet report, sending...
I20260619 14:50:50.673332 17971 ts_manager.cc:194] Registered new tserver with Master: 66c83651852747809a19d5295165b178 (127.0.98.2:39693)
I20260619 14:50:50.673972 17971 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:48609
W20260619 14:50:50.787442 18293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:50:50.787729 18293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:50:50.787768 18293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:50:50.792300 18293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:50:50.792412 18293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:50:50.792506 18293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 14:50:50.798044 18293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32857
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880383132139-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880383132139-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880383132139-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46749
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18293
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880383132139-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:50:50.799167 18293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:50:50.800550 18293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:50:50.811693 18299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:50.811725 18301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:50.811712 18298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:50:50.812212 18293 server_base.cc:1034] running on GCE node
I20260619 14:50:50.812775 18293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:50:50.813500 18293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:50:50.814700 18293 hybrid_clock.cc:648] HybridClock initialized: now 1781880650814664 us; error 50 us; skew 500 ppm
I20260619 14:50:50.817268 18293 webserver.cc:460] Webserver started at http://127.0.98.3:44149/ using document root <none> and password file <none>
I20260619 14:50:50.817920 18293 fs_manager.cc:359] Metadata directory not provided
I20260619 14:50:50.817988 18293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:50:50.818214 18293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:50:50.820770 18293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880383132139-392-0/minicluster-data/ts-2/data/instance:
uuid: "f9d98dd6d52a42fb808f11113d7133ba"
format_stamp: "Formatted at 2026-06-19 14:50:50 on dist-test-slave-2sb1"
server_key: "d4d79a22d02123f6ce682e5231bff1d6"
server_key_iv: "1365d26fef72a8067d6dd9c1eb975f70"
server_key_version: "encryptionkey@0"
I20260619 14:50:50.821452 18293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880383132139-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f9d98dd6d52a42fb808f11113d7133ba"
format_stamp: "Formatted at 2026-06-19 14:50:50 on dist-test-slave-2sb1"
server_key: "d4d79a22d02123f6ce682e5231bff1d6"
server_key_iv: "1365d26fef72a8067d6dd9c1eb975f70"
server_key_version: "encryptionkey@0"
I20260619 14:50:50.826092 18293 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260619 14:50:50.829226 18307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:50.830891 18293 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:50:50.831007 18293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880383132139-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880383132139-392-0/minicluster-data/ts-2/wal
uuid: "f9d98dd6d52a42fb808f11113d7133ba"
format_stamp: "Formatted at 2026-06-19 14:50:50 on dist-test-slave-2sb1"
server_key: "d4d79a22d02123f6ce682e5231bff1d6"
server_key_iv: "1365d26fef72a8067d6dd9c1eb975f70"
server_key_version: "encryptionkey@0"
I20260619 14:50:50.831173 18293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880383132139-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880383132139-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880383132139-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:50:50.855549 18293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:50:50.856359 18293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:50:50.856577 18293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:50:50.857220 18293 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:50:50.858158 18293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:50:50.858206 18293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:50.858245 18293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:50:50.858260 18293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:50.869356 18293 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34731
I20260619 14:50:50.869378 18420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34731 every 8 connection(s)
I20260619 14:50:50.870440 18293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880383132139-392-0/minicluster-data/ts-2/data/info.pb
I20260619 14:50:50.879096 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 18293
I20260619 14:50:50.879237 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880383132139-392-0/minicluster-data/ts-2/wal/instance
I20260619 14:50:50.879529 392 external_mini_cluster.cc:1428] Setting key fefdb008fa0b09dce44204781b95dbfc
I20260619 14:50:50.883105 18010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:50.872827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40881 (local address 127.0.98.62:46749)
0619 14:50:50.873027 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:50.873033 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:50.874488 (+ 1455us) server_negotiation.cc:407] Connection header received
0619 14:50:50.876458 (+ 1970us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:50.876462 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:50.876517 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:50.876604 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:50.877863 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:50.878586 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:50.880340 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:50.880708 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:50.880828 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:50.882568 (+ 1740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:50.882571 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:50.882573 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:50:50.882586 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:50.882607 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:50:50.882615 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:50.882682 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:50.882917 (+ 235us) server_negotiation.cc:299] Negotiation successful
0619 14:50:50.882977 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260619 14:50:50.884269 18423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:50.873381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46749 (local address 127.0.98.3:40881)
0619 14:50:50.874294 (+ 913us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:50.874341 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:50.876195 (+ 1854us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:50.876864 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:50.876872 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:50.876968 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:50.877659 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:50.877675 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:50.878773 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:50.878779 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:50.880225 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:50.880236 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:50.880901 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:50.880906 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:50.881124 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:50.882131 (+ 1007us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:50:50.882132 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:50.882207 (+ 75us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:50:50.882303 (+ 96us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:50:50.882726 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:50:50.882734 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:50:50.882739 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 14:50:50.882883 (+ 144us) client_negotiation.cc:240] Negotiation successful
0619 14:50:50.883316 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":281,"threads_started":1}
I20260619 14:50:50.886334 18421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46749
I20260619 14:50:50.886780 18421 heartbeater.cc:464] Registering TS with master...
I20260619 14:50:50.887395 18421 heartbeater.cc:511] Master 127.0.98.62:46749 requested a full tablet report, sending...
I20260619 14:50:50.888834 17971 ts_manager.cc:194] Registered new tserver with Master: f9d98dd6d52a42fb808f11113d7133ba (127.0.98.3:34731)
I20260619 14:50:50.889550 17971 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:40881
I20260619 14:50:50.895190 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:50:50.902411 18010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:50.897952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36184 (local address 127.0.98.62:46749)
0619 14:50:50.898152 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:50.898157 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:50.898285 (+ 128us) server_negotiation.cc:407] Connection header received
0619 14:50:50.898629 (+ 344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:50.898632 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:50.898688 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:50.898782 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:50.899075 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:50.899603 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:50.900512 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:50.900816 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:50.900936 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:50.901886 (+ 950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:50.901893 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:50.901901 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:50:50.901917 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:50.901937 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:50:50.901946 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:50.902042 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:50.902184 (+ 142us) server_negotiation.cc:299] Negotiation successful
0619 14:50:50.902248 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260619 14:50:50.906991 17971 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36184:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:50:50.911250 17971 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:50:50.929535 18433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:50.923430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48632 (local address 127.0.98.1:33973)
0619 14:50:50.924268 (+ 838us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:50.924277 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:50.924308 (+ 31us) server_negotiation.cc:407] Connection header received
0619 14:50:50.924404 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:50.924410 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:50.924674 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:50.925218 (+ 544us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:50:50.925392 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:50.926595 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:50.928087 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:50.928789 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:50.928925 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:50.929142 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:50.929248 (+ 106us) server_negotiation.cc:299] Negotiation successful
0619 14:50:50.929352 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":106,"threads_started":1}
I20260619 14:50:50.929527 18432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:50.922908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33973 (local address 127.0.0.1:48632)
0619 14:50:50.923579 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:50.924089 (+ 510us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:50.924223 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:50.924854 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:50.924860 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:50.924884 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:50:50.925049 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:50.925056 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:50.926717 (+ 1661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:50.926720 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:50.927943 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:50.927955 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:50.928927 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:50.928930 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:50.929049 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:50.929087 (+ 38us) client_negotiation.cc:769] Sending connection context
0619 14:50:50.929217 (+ 130us) client_negotiation.cc:240] Negotiation successful
0619 14:50:50.929361 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":223,"threads_started":1}
I20260619 14:50:50.929556 18434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:50.923804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34731 (local address 127.0.0.1:45264)
0619 14:50:50.924122 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:50.924138 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:50.924240 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:50.925274 (+ 1034us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:50.925277 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:50.925295 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:50:50.925540 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:50.925549 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:50.926529 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:50.926535 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:50.928001 (+ 1466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:50.928010 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:50.929142 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:50.929146 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:50.929248 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:50.929270 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 14:50:50.929322 (+ 52us) client_negotiation.cc:240] Negotiation successful
0619 14:50:50.929398 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":137,"threads_started":1}
I20260619 14:50:50.930151 18436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:50.923935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45264 (local address 127.0.98.3:34731)
0619 14:50:50.924868 (+ 933us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:50.924886 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:50.924911 (+ 25us) server_negotiation.cc:407] Connection header received
0619 14:50:50.924985 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:50.924989 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:50.925127 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:50.925303 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:50:50.925677 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:50.926409 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:50.928133 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:50.928990 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:50.929626 (+ 636us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:50.929833 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:50.929898 (+ 65us) server_negotiation.cc:299] Negotiation successful
0619 14:50:50.929994 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":813,"thread_start_us":128,"threads_started":1}
I20260619 14:50:50.930889 18435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:50.923832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39693 (local address 127.0.0.1:55616)
0619 14:50:50.924483 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:50.924496 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:50.924573 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:50.925364 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:50.925367 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:50.925385 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:50:50.925482 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:50.925487 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:50.926961 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:50.926965 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:50.928074 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:50.928084 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:50.929142 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:50.929146 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:50.929510 (+ 364us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:50.929531 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 14:50:50.930666 (+ 1135us) client_negotiation.cc:240] Negotiation successful
0619 14:50:50.930747 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"mutex_wait_us":275,"spinlock_wait_cycles":12672,"thread_start_us":118,"threads_started":1}
I20260619 14:50:50.931610 18437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:50.924374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55616 (local address 127.0.98.2:39693)
0619 14:50:50.924922 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:50.924933 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:50.924955 (+ 22us) server_negotiation.cc:407] Connection header received
0619 14:50:50.925026 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:50.925031 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:50.925215 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:50.925625 (+ 410us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:50:50.925762 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:50.926817 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:50.928208 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:50.929001 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:50.929194 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:50.929385 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:50.931343 (+ 1958us) server_negotiation.cc:299] Negotiation successful
0619 14:50:50.931453 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":97,"threads_started":1}
I20260619 14:50:50.932986 18093 tablet_service.cc:1467] Processing CreateTablet for tablet f5c923ead09f42b7a30bac363a07061e (DEFAULT_TABLE table=test-table [id=052ac09ef84b4bae979e3efa6a37b1f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:50:50.934216 18093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5c923ead09f42b7a30bac363a07061e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:50:50.933799 18355 tablet_service.cc:1467] Processing CreateTablet for tablet f5c923ead09f42b7a30bac363a07061e (DEFAULT_TABLE table=test-table [id=052ac09ef84b4bae979e3efa6a37b1f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:50:50.934815 18355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5c923ead09f42b7a30bac363a07061e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:50:50.934583 18222 tablet_service.cc:1467] Processing CreateTablet for tablet f5c923ead09f42b7a30bac363a07061e (DEFAULT_TABLE table=test-table [id=052ac09ef84b4bae979e3efa6a37b1f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:50:50.935491 18222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5c923ead09f42b7a30bac363a07061e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:50:50.942162 18439 tablet_bootstrap.cc:492] T f5c923ead09f42b7a30bac363a07061e P 4d18235f6f19499abee19b697dbbc993: Bootstrap starting.
I20260619 14:50:50.942665 18438 tablet_bootstrap.cc:492] T f5c923ead09f42b7a30bac363a07061e P f9d98dd6d52a42fb808f11113d7133ba: Bootstrap starting.
I20260619 14:50:50.943617 18440 tablet_bootstrap.cc:492] T f5c923ead09f42b7a30bac363a07061e P 66c83651852747809a19d5295165b178: Bootstrap starting.
I20260619 14:50:50.945779 18440 tablet_bootstrap.cc:654] T f5c923ead09f42b7a30bac363a07061e P 66c83651852747809a19d5295165b178: Neither blocks nor log segments found. Creating new log.
I20260619 14:50:50.945779 18439 tablet_bootstrap.cc:654] T f5c923ead09f42b7a30bac363a07061e P 4d18235f6f19499abee19b697dbbc993: Neither blocks nor log segments found. Creating new log.
I20260619 14:50:50.945773 18438 tablet_bootstrap.cc:654] T f5c923ead09f42b7a30bac363a07061e P f9d98dd6d52a42fb808f11113d7133ba: Neither blocks nor log segments found. Creating new log.
I20260619 14:50:50.946705 18438 log.cc:824] T f5c923ead09f42b7a30bac363a07061e P f9d98dd6d52a42fb808f11113d7133ba: Log is configured to *not* fsync() on all Append() calls
I20260619 14:50:50.946715 18439 log.cc:824] T f5c923ead09f42b7a30bac363a07061e P 4d18235f6f19499abee19b697dbbc993: Log is configured to *not* fsync() on all Append() calls
I20260619 14:50:50.946715 18440 log.cc:824] T f5c923ead09f42b7a30bac363a07061e P 66c83651852747809a19d5295165b178: Log is configured to *not* fsync() on all Append() calls
I20260619 14:50:50.948899 18440 tablet_bootstrap.cc:492] T f5c923ead09f42b7a30bac363a07061e P 66c83651852747809a19d5295165b178: No bootstrap required, opened a new log
I20260619 14:50:50.949077 18439 tablet_bootstrap.cc:492] T f5c923ead09f42b7a30bac363a07061e P 4d18235f6f19499abee19b697dbbc993: No bootstrap required, opened a new log
I20260619 14:50:50.949159 18440 ts_tablet_manager.cc:1397] T f5c923ead09f42b7a30bac363a07061e P 66c83651852747809a19d5295165b178: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260619 14:50:50.949252 18438 tablet_bootstrap.cc:492] T f5c923ead09f42b7a30bac363a07061e P f9d98dd6d52a42fb808f11113d7133ba: No bootstrap required, opened a new log
I20260619 14:50:50.949294 18439 ts_tablet_manager.cc:1397] T f5c923ead09f42b7a30bac363a07061e P 4d18235f6f19499abee19b697dbbc993: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260619 14:50:50.949435 18438 ts_tablet_manager.cc:1397] T f5c923ead09f42b7a30bac363a07061e P f9d98dd6d52a42fb808f11113d7133ba: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260619 14:50:50.953025 18440 raft_consensus.cc:348] T f5c923ead09f42b7a30bac363a07061e P 66c83651852747809a19d5295165b178 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d18235f6f19499abee19b697dbbc993" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33973 } } peers { permanent_uuid: "66c83651852747809a19d5295165b178" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39693 } } peers { permanent_uuid: "f9d98dd6d52a42fb808f11113d7133ba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34731 } }
I20260619 14:50:50.953306 18440 raft_consensus.cc:374] T f5c923ead09f42b7a30bac363a07061e P 66c83651852747809a19d5295165b178 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:50:50.953362 18440 raft_consensus.cc:729] T f5c923ead09f42b7a30bac363a07061e P 66c83651852747809a19d5295165b178 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66c83651852747809a19d5295165b178, State: Initialized, Role: FOLLOWER
I20260619 14:50:50.953825 18440 consensus_queue.cc:260] T f5c923ead09f42b7a30bac363a07061e P 66c83651852747809a19d5295165b178 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d18235f6f19499abee19b697dbbc993" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33973 } } peers { permanent_uuid: "66c83651852747809a19d5295165b178" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39693 } } peers { permanent_uuid: "f9d98dd6d52a42fb808f11113d7133ba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34731 } }
I20260619 14:50:50.954205 18439 raft_consensus.cc:348] T f5c923ead09f42b7a30bac363a07061e P 4d18235f6f19499abee19b697dbbc993 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d18235f6f19499abee19b697dbbc993" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33973 } } peers { permanent_uuid: "66c83651852747809a19d5295165b178" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39693 } } peers { permanent_uuid: "f9d98dd6d52a42fb808f11113d7133ba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34731 } }
I20260619 14:50:50.954205 18438 raft_consensus.cc:348] T f5c923ead09f42b7a30bac363a07061e P f9d98dd6d52a42fb808f11113d7133ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d18235f6f19499abee19b697dbbc993" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33973 } } peers { permanent_uuid: "66c83651852747809a19d5295165b178" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39693 } } peers { permanent_uuid: "f9d98dd6d52a42fb808f11113d7133ba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34731 } }
I20260619 14:50:50.954494 18439 raft_consensus.cc:374] T f5c923ead09f42b7a30bac363a07061e P 4d18235f6f19499abee19b697dbbc993 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:50:50.954494 18438 raft_consensus.cc:374] T f5c923ead09f42b7a30bac363a07061e P f9d98dd6d52a42fb808f11113d7133ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:50:50.954543 18290 heartbeater.cc:503] Master 127.0.98.62:46749 was elected leader, sending a full tablet report...
I20260619 14:50:50.954550 18439 raft_consensus.cc:729] T f5c923ead09f42b7a30bac363a07061e P 4d18235f6f19499abee19b697dbbc993 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d18235f6f19499abee19b697dbbc993, State: Initialized, Role: FOLLOWER
I20260619 14:50:50.954550 18438 raft_consensus.cc:729] T f5c923ead09f42b7a30bac363a07061e P f9d98dd6d52a42fb808f11113d7133ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9d98dd6d52a42fb808f11113d7133ba, State: Initialized, Role: FOLLOWER
I20260619 14:50:50.954937 18440 ts_tablet_manager.cc:1428] T f5c923ead09f42b7a30bac363a07061e P 66c83651852747809a19d5295165b178: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260619 14:50:50.955159 18438 consensus_queue.cc:260] T f5c923ead09f42b7a30bac363a07061e P f9d98dd6d52a42fb808f11113d7133ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d18235f6f19499abee19b697dbbc993" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33973 } } peers { permanent_uuid: "66c83651852747809a19d5295165b178" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39693 } } peers { permanent_uuid: "f9d98dd6d52a42fb808f11113d7133ba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34731 } }
I20260619 14:50:50.955201 18439 consensus_queue.cc:260] T f5c923ead09f42b7a30bac363a07061e P 4d18235f6f19499abee19b697dbbc993 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d18235f6f19499abee19b697dbbc993" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33973 } } peers { permanent_uuid: "66c83651852747809a19d5295165b178" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39693 } } peers { permanent_uuid: "f9d98dd6d52a42fb808f11113d7133ba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34731 } }
I20260619 14:50:50.956437 18438 ts_tablet_manager.cc:1428] T f5c923ead09f42b7a30bac363a07061e P f9d98dd6d52a42fb808f11113d7133ba: Time spent starting tablet: real 0.007s user 0.003s sys 0.004s
I20260619 14:50:50.956437 18439 ts_tablet_manager.cc:1428] T f5c923ead09f42b7a30bac363a07061e P 4d18235f6f19499abee19b697dbbc993: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260619 14:50:50.956820 18159 heartbeater.cc:503] Master 127.0.98.62:46749 was elected leader, sending a full tablet report...
I20260619 14:50:50.956820 18421 heartbeater.cc:503] Master 127.0.98.62:46749 was elected leader, sending a full tablet report...
I20260619 14:50:51.095778 18446 raft_consensus.cc:482] T f5c923ead09f42b7a30bac363a07061e P 4d18235f6f19499abee19b697dbbc993 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:50:51.096036 18446 raft_consensus.cc:504] T f5c923ead09f42b7a30bac363a07061e P 4d18235f6f19499abee19b697dbbc993 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d18235f6f19499abee19b697dbbc993" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33973 } } peers { permanent_uuid: "66c83651852747809a19d5295165b178" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39693 } } peers { permanent_uuid: "f9d98dd6d52a42fb808f11113d7133ba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34731 } }
I20260619 14:50:51.097496 18446 leader_election.cc:288] T f5c923ead09f42b7a30bac363a07061e P 4d18235f6f19499abee19b697dbbc993 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 66c83651852747809a19d5295165b178 (127.0.98.2:39693), f9d98dd6d52a42fb808f11113d7133ba (127.0.98.3:34731)
I20260619 14:50:51.102510 18447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:51.097700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39693 (local address 127.0.98.1:57783)
0619 14:50:51.097999 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:51.098016 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:51.098137 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:51.098501 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:51.098506 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:51.098595 (+ 89us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:50:51.098754 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:51.098765 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:51.099771 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:51.099775 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:51.100943 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:51.100977 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:51.101929 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:51.101933 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:51.102068 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:51.102097 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 14:50:51.102186 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 14:50:51.102272 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"spinlock_wait_cycles":1408,"thread_start_us":133,"threads_started":1}
I20260619 14:50:51.102676 18437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:51.097917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57783 (local address 127.0.98.2:39693)
0619 14:50:51.098078 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:51.098083 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:51.098101 (+ 18us) server_negotiation.cc:407] Connection header received
0619 14:50:51.098275 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:51.098278 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:51.098349 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:51.098433 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:50:51.098960 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:51.099642 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:51.101105 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:51.101695 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:51.101787 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:51.101851 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:51.102237 (+ 386us) server_negotiation.cc:299] Negotiation successful
0619 14:50:51.102336 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 14:50:51.102712 18436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:51.098179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38979 (local address 127.0.98.3:34731)
0619 14:50:51.098325 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:51.098330 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:51.098612 (+ 282us) server_negotiation.cc:407] Connection header received
0619 14:50:51.098683 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:51.098686 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:51.098739 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:51.098850 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:50:51.099264 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:51.099950 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:51.101267 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:51.101957 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:51.102090 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:51.102158 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:51.102472 (+ 314us) server_negotiation.cc:299] Negotiation successful
0619 14:50:51.102578 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 14:50:51.102742 18448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:51.097945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34731 (local address 127.0.98.1:38979)
0619 14:50:51.098426 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:51.098439 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:51.098555 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:51.098955 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:51.098958 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:51.098979 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:50:51.099106 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:51.099112 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:51.100144 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:51.100147 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:51.101110 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:51.101121 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:51.102146 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:51.102181 (+ 35us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:51.102336 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:51.102362 (+ 26us) client_negotiation.cc:769] Sending connection context
0619 14:50:51.102455 (+ 93us) client_negotiation.cc:240] Negotiation successful
0619 14:50:51.102535 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"mutex_wait_us":1,"spinlock_wait_cycles":30848,"thread_start_us":140,"threads_started":1}
I20260619 14:50:51.103506 18375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5c923ead09f42b7a30bac363a07061e" candidate_uuid: "4d18235f6f19499abee19b697dbbc993" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9d98dd6d52a42fb808f11113d7133ba" is_pre_election: true
I20260619 14:50:51.103559 18244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5c923ead09f42b7a30bac363a07061e" candidate_uuid: "4d18235f6f19499abee19b697dbbc993" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66c83651852747809a19d5295165b178" is_pre_election: true
I20260619 14:50:51.103884 18375 raft_consensus.cc:2454] T f5c923ead09f42b7a30bac363a07061e P f9d98dd6d52a42fb808f11113d7133ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d18235f6f19499abee19b697dbbc993 in term 0.
I20260619 14:50:51.103879 18244 raft_consensus.cc:2454] T f5c923ead09f42b7a30bac363a07061e P 66c83651852747809a19d5295165b178 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d18235f6f19499abee19b697dbbc993 in term 0.
I20260619 14:50:51.104538 18049 leader_election.cc:302] T f5c923ead09f42b7a30bac363a07061e P 4d18235f6f19499abee19b697dbbc993 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4d18235f6f19499abee19b697dbbc993, 66c83651852747809a19d5295165b178, f9d98dd6d52a42fb808f11113d7133ba; no voters:
I20260619 14:50:51.104941 18446 raft_consensus.cc:2781] T f5c923ead09f42b7a30bac363a07061e P 4d18235f6f19499abee19b697dbbc993 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:50:51.105037 18446 raft_consensus.cc:482] T f5c923ead09f42b7a30bac363a07061e P 4d18235f6f19499abee19b697dbbc993 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:50:51.105098 18446 raft_consensus.cc:3037] T f5c923ead09f42b7a30bac363a07061e P 4d18235f6f19499abee19b697dbbc993 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:50:51.106606 18446 raft_consensus.cc:504] T f5c923ead09f42b7a30bac363a07061e P 4d18235f6f19499abee19b697dbbc993 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d18235f6f19499abee19b697dbbc993" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33973 } } peers { permanent_uuid: "66c83651852747809a19d5295165b178" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39693 } } peers { permanent_uuid: "f9d98dd6d52a42fb808f11113d7133ba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34731 } }
I20260619 14:50:51.107069 18446 leader_election.cc:288] T f5c923ead09f42b7a30bac363a07061e P 4d18235f6f19499abee19b697dbbc993 [CANDIDATE]: Term 1 election: Requested vote from peers 66c83651852747809a19d5295165b178 (127.0.98.2:39693), f9d98dd6d52a42fb808f11113d7133ba (127.0.98.3:34731)
I20260619 14:50:51.107705 18375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5c923ead09f42b7a30bac363a07061e" candidate_uuid: "4d18235f6f19499abee19b697dbbc993" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9d98dd6d52a42fb808f11113d7133ba"
I20260619 14:50:51.107694 18244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5c923ead09f42b7a30bac363a07061e" candidate_uuid: "4d18235f6f19499abee19b697dbbc993" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66c83651852747809a19d5295165b178"
I20260619 14:50:51.107956 18375 raft_consensus.cc:3037] T f5c923ead09f42b7a30bac363a07061e P f9d98dd6d52a42fb808f11113d7133ba [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:50:51.107956 18244 raft_consensus.cc:3037] T f5c923ead09f42b7a30bac363a07061e P 66c83651852747809a19d5295165b178 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:50:51.109524 18375 raft_consensus.cc:2454] T f5c923ead09f42b7a30bac363a07061e P f9d98dd6d52a42fb808f11113d7133ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d18235f6f19499abee19b697dbbc993 in term 1.
I20260619 14:50:51.109524 18244 raft_consensus.cc:2454] T f5c923ead09f42b7a30bac363a07061e P 66c83651852747809a19d5295165b178 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d18235f6f19499abee19b697dbbc993 in term 1.
I20260619 14:50:51.110006 18048 leader_election.cc:302] T f5c923ead09f42b7a30bac363a07061e P 4d18235f6f19499abee19b697dbbc993 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d18235f6f19499abee19b697dbbc993, 66c83651852747809a19d5295165b178; no voters:
I20260619 14:50:51.110224 18446 raft_consensus.cc:2781] T f5c923ead09f42b7a30bac363a07061e P 4d18235f6f19499abee19b697dbbc993 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:50:51.110517 18446 raft_consensus.cc:686] T f5c923ead09f42b7a30bac363a07061e P 4d18235f6f19499abee19b697dbbc993 [term 1 LEADER]: Becoming Leader. State: Replica: 4d18235f6f19499abee19b697dbbc993, State: Running, Role: LEADER
I20260619 14:50:51.110967 18446 consensus_queue.cc:237] T f5c923ead09f42b7a30bac363a07061e P 4d18235f6f19499abee19b697dbbc993 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d18235f6f19499abee19b697dbbc993" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33973 } } peers { permanent_uuid: "66c83651852747809a19d5295165b178" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39693 } } peers { permanent_uuid: "f9d98dd6d52a42fb808f11113d7133ba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34731 } }
I20260619 14:50:51.114928 17966 catalog_manager.cc:5495] T f5c923ead09f42b7a30bac363a07061e P 4d18235f6f19499abee19b697dbbc993 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d18235f6f19499abee19b697dbbc993 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "4d18235f6f19499abee19b697dbbc993" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d18235f6f19499abee19b697dbbc993" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33973 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "66c83651852747809a19d5295165b178" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39693 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9d98dd6d52a42fb808f11113d7133ba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34731 } health_report { overall_health: UNKNOWN } } }
W20260619 14:50:51.121649 18422 tablet.cc:2369] T f5c923ead09f42b7a30bac363a07061e P f9d98dd6d52a42fb808f11113d7133ba: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:50:51.132023 18433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:51.127840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48644 (local address 127.0.98.1:33973)
0619 14:50:51.127992 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:51.127996 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:51.128010 (+ 14us) server_negotiation.cc:407] Connection header received
0619 14:50:51.128088 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:51.128090 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:51.128147 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:51.128248 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:50:51.128547 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:51.129279 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:51.130346 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:51.130633 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:51.130732 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:51.131075 (+ 343us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:50:51.131543 (+ 468us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:50:51.131645 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:51.131812 (+ 167us) server_negotiation.cc:299] Negotiation successful
0619 14:50:51.131867 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 14:50:51.139189 18375 raft_consensus.cc:1261] T f5c923ead09f42b7a30bac363a07061e P f9d98dd6d52a42fb808f11113d7133ba [term 1 FOLLOWER]: Refusing update from remote peer 4d18235f6f19499abee19b697dbbc993: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:50:51.139190 18244 raft_consensus.cc:1261] T f5c923ead09f42b7a30bac363a07061e P 66c83651852747809a19d5295165b178 [term 1 FOLLOWER]: Refusing update from remote peer 4d18235f6f19499abee19b697dbbc993: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:50:51.140074 18449 consensus_queue.cc:1035] T f5c923ead09f42b7a30bac363a07061e P 4d18235f6f19499abee19b697dbbc993 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66c83651852747809a19d5295165b178" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:50:51.140319 18446 consensus_queue.cc:1035] T f5c923ead09f42b7a30bac363a07061e P 4d18235f6f19499abee19b697dbbc993 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9d98dd6d52a42fb808f11113d7133ba" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:50:51.148969 18454 mvcc.cc:205] Tried to move back new op lower bound from 7298583147056693248 to 7298583146954096640. Current Snapshot: MvccSnapshot[applied={T|T < 7298583147056693248}]
I20260619 14:50:51.158602 17966 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36184:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:50:51.158793 17966 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36184:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:50:51.161638 17966 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 66644343d480449d9ae2168c2dc7f97c: Sending DeleteTablet for 3 replicas of tablet f5c923ead09f42b7a30bac363a07061e
I20260619 14:50:51.162761 18093 tablet_service.cc:1514] Processing DeleteTablet for tablet f5c923ead09f42b7a30bac363a07061e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:50:51 UTC) from {username='slave'} at 127.0.0.1:48632
I20260619 14:50:51.162892 18355 tablet_service.cc:1514] Processing DeleteTablet for tablet f5c923ead09f42b7a30bac363a07061e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:50:51 UTC) from {username='slave'} at 127.0.0.1:45264
I20260619 14:50:51.163412 18222 tablet_service.cc:1514] Processing DeleteTablet for tablet f5c923ead09f42b7a30bac363a07061e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:50:51 UTC) from {username='slave'} at 127.0.0.1:55616
I20260619 14:50:51.164228 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 18031
I20260619 14:50:51.165313 18463 tablet_replica.cc:330] T f5c923ead09f42b7a30bac363a07061e P 66c83651852747809a19d5295165b178: stopping tablet replica
I20260619 14:50:51.165717 18463 raft_consensus.cc:2229] T f5c923ead09f42b7a30bac363a07061e P 66c83651852747809a19d5295165b178 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:50:51.166137 18463 raft_consensus.cc:2258] T f5c923ead09f42b7a30bac363a07061e P 66c83651852747809a19d5295165b178 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:50:51.166237 18462 tablet_replica.cc:330] T f5c923ead09f42b7a30bac363a07061e P f9d98dd6d52a42fb808f11113d7133ba: stopping tablet replica
I20260619 14:50:51.166713 18462 raft_consensus.cc:2229] T f5c923ead09f42b7a30bac363a07061e P f9d98dd6d52a42fb808f11113d7133ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:50:51.167292 18462 raft_consensus.cc:2258] T f5c923ead09f42b7a30bac363a07061e P f9d98dd6d52a42fb808f11113d7133ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:50:51.168337 18463 ts_tablet_manager.cc:1905] T f5c923ead09f42b7a30bac363a07061e P 66c83651852747809a19d5295165b178: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:50:51.169034 18462 ts_tablet_manager.cc:1905] T f5c923ead09f42b7a30bac363a07061e P f9d98dd6d52a42fb808f11113d7133ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:50:51.171397 18463 ts_tablet_manager.cc:1918] T f5c923ead09f42b7a30bac363a07061e P 66c83651852747809a19d5295165b178: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:50:51.171614 18463 log.cc:1196] T f5c923ead09f42b7a30bac363a07061e P 66c83651852747809a19d5295165b178: Deleting WAL directory at /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880383132139-392-0/minicluster-data/ts-1/wal/wals/f5c923ead09f42b7a30bac363a07061e
I20260619 14:50:51.172000 18463 ts_tablet_manager.cc:1939] T f5c923ead09f42b7a30bac363a07061e P 66c83651852747809a19d5295165b178: Deleting consensus metadata
I20260619 14:50:51.172588 18462 ts_tablet_manager.cc:1918] T f5c923ead09f42b7a30bac363a07061e P f9d98dd6d52a42fb808f11113d7133ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:50:51.172727 18462 log.cc:1196] T f5c923ead09f42b7a30bac363a07061e P f9d98dd6d52a42fb808f11113d7133ba: Deleting WAL directory at /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781880383132139-392-0/minicluster-data/ts-2/wal/wals/f5c923ead09f42b7a30bac363a07061e
I20260619 14:50:51.173072 18462 ts_tablet_manager.cc:1939] T f5c923ead09f42b7a30bac363a07061e P f9d98dd6d52a42fb808f11113d7133ba: Deleting consensus metadata
W20260619 14:50:51.173449 17955 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33973 (error 108)
I20260619 14:50:51.173806 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 18162
I20260619 14:50:51.176208 18435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:51.175772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33973 (local address 127.0.0.1:48646)
0619 14:50:51.175996 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:51.176092 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
W20260619 14:50:51.176437 17955 catalog_manager.cc:4584] TS 4d18235f6f19499abee19b697dbbc993 (127.0.98.1:33973): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f5c923ead09f42b7a30bac363a07061e: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33973: connect: Connection refused (error 111)
I20260619 14:50:51.177043 17958 catalog_manager.cc:4841] TS f9d98dd6d52a42fb808f11113d7133ba (127.0.98.3:34731): tablet f5c923ead09f42b7a30bac363a07061e (table test-table [id=052ac09ef84b4bae979e3efa6a37b1f9]) successfully deleted
I20260619 14:50:51.180549 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 18293
I20260619 14:50:51.181854 18435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:51.181453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39693 (local address 127.0.0.1:55618)
0619 14:50:51.181677 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:51.181741 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:39693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
I20260619 14:50:51.186318 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 17940
2026-06-19T14:50:51Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1276 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880383132139-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:50:51 dist-test-slave-2sb1 krb5kdc[18467](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:50:51 dist-test-slave-2sb1 krb5kdc[18467](info): set up 2 sockets
Jun 19 14:50:51 dist-test-slave-2sb1 krb5kdc[18467](info): commencing operation
krb5kdc: starting...
W20260619 14:50:52.281643 392 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880383132139-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880383132139-392-0/krb5kdc/test-user.keytab.
Jun 19 14:50:52 dist-test-slave-2sb1 krb5kdc[18467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:50:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:50:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 14:50:52.429383 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880383132139-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880383132139-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880383132139-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36847
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36847
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:50:52.550247 18480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:50:52.550619 18480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:50:52.550714 18480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:50:52.555366 18480 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:50:52.555498 18480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:50:52.555533 18480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:50:52.555595 18480 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:50:52.555626 18480 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:50:52.561939 18480 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880383132139-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880383132139-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36847
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36847
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18480
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880383132139-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:50:52.563423 18480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:50:52.564901 18480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:50:52.575213 18488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:52.575213 18486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:52.575213 18485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:50:52.575264 18480 server_base.cc:1034] running on GCE node
I20260619 14:50:52.576345 18480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:50:52.577822 18480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:50:52.579061 18480 hybrid_clock.cc:648] HybridClock initialized: now 1781880652579044 us; error 66 us; skew 500 ppm
Jun 19 14:50:52 dist-test-slave-2sb1 krb5kdc[18467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880652, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:50:52.589884 18480 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 14:50:52.591198 18480 webserver.cc:460] Webserver started at http://127.0.98.62:36113/ using document root <none> and password file <none>
I20260619 14:50:52.591835 18480 fs_manager.cc:359] Metadata directory not provided
I20260619 14:50:52.591945 18480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:50:52.592191 18480 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:50:52.594540 18480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880383132139-392-0/minicluster-data/master-0/data/instance:
uuid: "b94d059e3a1844c59e878b66a2f18ca2"
format_stamp: "Formatted at 2026-06-19 14:50:52 on dist-test-slave-2sb1"
server_key: "03afd67fc05829097539390c8eecc010"
server_key_iv: "a9e5db9c368a8f7f8770ef79bb59358c"
server_key_version: "encryptionkey@0"
I20260619 14:50:52.595180 18480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880383132139-392-0/minicluster-data/master-0/wal/instance:
uuid: "b94d059e3a1844c59e878b66a2f18ca2"
format_stamp: "Formatted at 2026-06-19 14:50:52 on dist-test-slave-2sb1"
server_key: "03afd67fc05829097539390c8eecc010"
server_key_iv: "a9e5db9c368a8f7f8770ef79bb59358c"
server_key_version: "encryptionkey@0"
I20260619 14:50:52.599797 18480 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260619 14:50:52.603226 18495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:52.605314 18480 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 14:50:52.605526 18480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880383132139-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880383132139-392-0/minicluster-data/master-0/wal
uuid: "b94d059e3a1844c59e878b66a2f18ca2"
format_stamp: "Formatted at 2026-06-19 14:50:52 on dist-test-slave-2sb1"
server_key: "03afd67fc05829097539390c8eecc010"
server_key_iv: "a9e5db9c368a8f7f8770ef79bb59358c"
server_key_version: "encryptionkey@0"
I20260619 14:50:52.605712 18480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880383132139-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880383132139-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880383132139-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:50:52.626159 18480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:50:52.629859 18480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:50:52.630172 18480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:50:52.638165 18480 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36847
I20260619 14:50:52.638196 18547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36847 every 8 connection(s)
I20260619 14:50:52.639298 18480 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:50:52.642669 18548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:50:52.647869 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 18480
I20260619 14:50:52.648075 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880383132139-392-0/minicluster-data/master-0/wal/instance
I20260619 14:50:52.648416 392 external_mini_cluster.cc:1428] Setting key 2985fc55ea7203235f131326a4c6ea3a
I20260619 14:50:52.649777 18548 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b94d059e3a1844c59e878b66a2f18ca2: Bootstrap starting.
I20260619 14:50:52.652374 18548 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b94d059e3a1844c59e878b66a2f18ca2: Neither blocks nor log segments found. Creating new log.
I20260619 14:50:52.653275 18548 log.cc:824] T 00000000000000000000000000000000 P b94d059e3a1844c59e878b66a2f18ca2: Log is configured to *not* fsync() on all Append() calls
I20260619 14:50:52.655352 18548 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b94d059e3a1844c59e878b66a2f18ca2: No bootstrap required, opened a new log
Jun 19 14:50:52 dist-test-slave-2sb1 krb5kdc[18467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:50:52.659902 18548 raft_consensus.cc:348] T 00000000000000000000000000000000 P b94d059e3a1844c59e878b66a2f18ca2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b94d059e3a1844c59e878b66a2f18ca2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36847 } }
I20260619 14:50:52.660277 18548 raft_consensus.cc:374] T 00000000000000000000000000000000 P b94d059e3a1844c59e878b66a2f18ca2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:50:52.660372 18548 raft_consensus.cc:729] T 00000000000000000000000000000000 P b94d059e3a1844c59e878b66a2f18ca2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b94d059e3a1844c59e878b66a2f18ca2, State: Initialized, Role: FOLLOWER
I20260619 14:50:52.661015 18548 consensus_queue.cc:260] T 00000000000000000000000000000000 P b94d059e3a1844c59e878b66a2f18ca2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b94d059e3a1844c59e878b66a2f18ca2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36847 } }
I20260619 14:50:52.661203 18548 raft_consensus.cc:388] T 00000000000000000000000000000000 P b94d059e3a1844c59e878b66a2f18ca2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:50:52.661298 18548 raft_consensus.cc:482] T 00000000000000000000000000000000 P b94d059e3a1844c59e878b66a2f18ca2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:50:52.661394 18548 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b94d059e3a1844c59e878b66a2f18ca2 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:50:52.662384 18548 raft_consensus.cc:504] T 00000000000000000000000000000000 P b94d059e3a1844c59e878b66a2f18ca2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b94d059e3a1844c59e878b66a2f18ca2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36847 } }
I20260619 14:50:52.662756 18548 leader_election.cc:302] T 00000000000000000000000000000000 P b94d059e3a1844c59e878b66a2f18ca2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b94d059e3a1844c59e878b66a2f18ca2; no voters:
I20260619 14:50:52.663125 18548 leader_election.cc:288] T 00000000000000000000000000000000 P b94d059e3a1844c59e878b66a2f18ca2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:50:52.663275 18553 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b94d059e3a1844c59e878b66a2f18ca2 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:50:52.663573 18553 raft_consensus.cc:686] T 00000000000000000000000000000000 P b94d059e3a1844c59e878b66a2f18ca2 [term 1 LEADER]: Becoming Leader. State: Replica: b94d059e3a1844c59e878b66a2f18ca2, State: Running, Role: LEADER
I20260619 14:50:52.663932 18553 consensus_queue.cc:237] T 00000000000000000000000000000000 P b94d059e3a1844c59e878b66a2f18ca2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b94d059e3a1844c59e878b66a2f18ca2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36847 } }
I20260619 14:50:52.664621 18548 sys_catalog.cc:564] T 00000000000000000000000000000000 P b94d059e3a1844c59e878b66a2f18ca2 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:50:52.665788 18554 sys_catalog.cc:455] T 00000000000000000000000000000000 P b94d059e3a1844c59e878b66a2f18ca2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b94d059e3a1844c59e878b66a2f18ca2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b94d059e3a1844c59e878b66a2f18ca2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36847 } } }
I20260619 14:50:52.665942 18554 sys_catalog.cc:458] T 00000000000000000000000000000000 P b94d059e3a1844c59e878b66a2f18ca2 [sys.catalog]: This master's current role is: LEADER
I20260619 14:50:52.666479 18551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:52.650112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36462 (local address 127.0.98.62:36847)
0619 14:50:52.650723 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:52.650741 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:52.650782 (+ 41us) server_negotiation.cc:407] Connection header received
0619 14:50:52.651585 (+ 803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:52.651607 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:52.652013 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:52.652456 (+ 443us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:52.653456 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:52.654542 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:52.655570 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:52.656266 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:52.656470 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:52.659850 (+ 3380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:52.659858 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:52.659876 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:52.659916 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:52.662693 (+ 2777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:52.663381 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:52.663390 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:52.663396 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:52.663549 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:52.663868 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:52.663871 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:52.663873 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:52.664431 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:52.664670 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:52.664979 (+ 309us) server_negotiation.cc:299] Negotiation successful
0619 14:50:52.665314 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":157,"threads_started":1}
I20260619 14:50:52.666507 18555 sys_catalog.cc:455] T 00000000000000000000000000000000 P b94d059e3a1844c59e878b66a2f18ca2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b94d059e3a1844c59e878b66a2f18ca2. Latest consensus state: current_term: 1 leader_uuid: "b94d059e3a1844c59e878b66a2f18ca2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b94d059e3a1844c59e878b66a2f18ca2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36847 } } }
I20260619 14:50:52.666621 18555 sys_catalog.cc:458] T 00000000000000000000000000000000 P b94d059e3a1844c59e878b66a2f18ca2 [sys.catalog]: This master's current role is: LEADER
W20260619 14:50:52.669680 18568 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b94d059e3a1844c59e878b66a2f18ca2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:50:52.669761 18568 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:50:52.669859 18562 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:50:52.670629 18562 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:50:52.676120 18562 catalog_manager.cc:1347] Generated new cluster ID: 48de0374f2ae43168868ee08ee86d6ba
I20260619 14:50:52.676214 18562 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:50:52.691154 18562 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:50:52.692543 18562 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:50:52.701324 18562 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b94d059e3a1844c59e878b66a2f18ca2: Generated new TSK 0
I20260619 14:50:52.702338 18562 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880383132139-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880383132139-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 14:50:52.744889 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880383132139-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880383132139-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880383132139-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880383132139-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36847
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880383132139-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:50:52.867165 18574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:50:52.867592 18574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:50:52.867684 18574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:50:52.872109 18574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:50:52.872218 18574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:50:52.872320 18574 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 14:50:52.878291 18574 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880383132139-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880383132139-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880383132139-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880383132139-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36847
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18574
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880383132139-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:50:52.879654 18574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:50:52.880764 18574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:50:52.892377 18580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:52.892366 18582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:52.892346 18579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:50:52.892899 18574 server_base.cc:1034] running on GCE node
I20260619 14:50:52.893374 18574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:50:52.894203 18574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:50:52.895419 18574 hybrid_clock.cc:648] HybridClock initialized: now 1781880652895397 us; error 37 us; skew 500 ppm
Jun 19 14:50:52 dist-test-slave-2sb1 krb5kdc[18467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880652, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:50:52.898802 18574 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 14:50:52.899859 18574 webserver.cc:460] Webserver started at http://127.0.98.1:34301/ using document root <none> and password file <none>
I20260619 14:50:52.900573 18574 fs_manager.cc:359] Metadata directory not provided
I20260619 14:50:52.900698 18574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:50:52.900990 18574 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:50:52.903357 18574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880383132139-392-0/minicluster-data/ts-0/data/instance:
uuid: "1630bb912ac143ef9bfa21bc3546e444"
format_stamp: "Formatted at 2026-06-19 14:50:52 on dist-test-slave-2sb1"
server_key: "08233c4436e041713af6494750789bd3"
server_key_iv: "209d6e3ba8c60ea2c47bc18f7f99653d"
server_key_version: "encryptionkey@0"
I20260619 14:50:52.903975 18574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880383132139-392-0/minicluster-data/ts-0/wal/instance:
uuid: "1630bb912ac143ef9bfa21bc3546e444"
format_stamp: "Formatted at 2026-06-19 14:50:52 on dist-test-slave-2sb1"
server_key: "08233c4436e041713af6494750789bd3"
server_key_iv: "209d6e3ba8c60ea2c47bc18f7f99653d"
server_key_version: "encryptionkey@0"
I20260619 14:50:52.908526 18574 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 14:50:52.912400 18589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:52.914248 18574 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:50:52.914376 18574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880383132139-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880383132139-392-0/minicluster-data/ts-0/wal
uuid: "1630bb912ac143ef9bfa21bc3546e444"
format_stamp: "Formatted at 2026-06-19 14:50:52 on dist-test-slave-2sb1"
server_key: "08233c4436e041713af6494750789bd3"
server_key_iv: "209d6e3ba8c60ea2c47bc18f7f99653d"
server_key_version: "encryptionkey@0"
I20260619 14:50:52.914491 18574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880383132139-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880383132139-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880383132139-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:50:52.931527 18574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:50:52.935390 18574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:50:52.935580 18574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:50:52.936167 18574 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:50:52.937110 18574 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:50:52.937172 18574 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:52.937216 18574 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:50:52.937237 18574 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:52.947944 18574 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43603
I20260619 14:50:52.948052 18702 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43603 every 8 connection(s)
I20260619 14:50:52.949129 18574 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880383132139-392-0/minicluster-data/ts-0/data/info.pb
I20260619 14:50:52.953317 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 18574
I20260619 14:50:52.953495 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880383132139-392-0/minicluster-data/ts-0/wal/instance
I20260619 14:50:52.953811 392 external_mini_cluster.cc:1428] Setting key 2209166e1cca6b5b10dc636d7a52b1f9
Jun 19 14:50:52 dist-test-slave-2sb1 krb5kdc[18467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880652, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 14:50:52.968312 18551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:52.951264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50907 (local address 127.0.98.62:36847)
0619 14:50:52.951443 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:52.951447 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:52.952281 (+ 834us) server_negotiation.cc:407] Connection header received
0619 14:50:52.954150 (+ 1869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:52.954154 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:52.954212 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:52.954332 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:52.956035 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:52.956612 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:52.958807 (+ 2195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:52.959229 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:52.959429 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:52.963352 (+ 3923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:52.963355 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:52.963357 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:52.963386 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:52.965226 (+ 1840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:52.965953 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:52.965957 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:52.965959 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:52.966055 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:52.966665 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:52.966668 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:52.966669 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:52.966897 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:52.967159 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:52.967934 (+ 775us) server_negotiation.cc:299] Negotiation successful
0619 14:50:52.968103 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260619 14:50:52.969446 18705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:52.951557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36847 (local address 127.0.98.1:50907)
0619 14:50:52.952091 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:52.952143 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:52.953758 (+ 1615us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:52.954571 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:52.954604 (+ 33us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:52.955170 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:52.955850 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:52.955867 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:52.956797 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:52.956803 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:52.958656 (+ 1853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:52.958670 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:52.959493 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:52.959497 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:52.959692 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:52.960753 (+ 1061us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:52.960780 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:52.963112 (+ 2332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:50:52.965382 (+ 2270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:52.965391 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:52.965412 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:52.965810 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:52.966179 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:52.966182 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:52.966185 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:52.966374 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:52.967153 (+ 779us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:50:52.967162 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:50:52.967587 (+ 425us) client_negotiation.cc:769] Sending connection context
0619 14:50:52.967882 (+ 295us) client_negotiation.cc:240] Negotiation successful
0619 14:50:52.968332 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":134,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 14:50:52.971336 18703 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36847
I20260619 14:50:52.971603 18703 heartbeater.cc:464] Registering TS with master...
I20260619 14:50:52.972179 18703 heartbeater.cc:511] Master 127.0.98.62:36847 requested a full tablet report, sending...
I20260619 14:50:52.973862 18512 ts_manager.cc:194] Registered new tserver with Master: 1630bb912ac143ef9bfa21bc3546e444 (127.0.98.1:43603)
I20260619 14:50:52.975517 18512 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:50907
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880383132139-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880383132139-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 14:50:52.989092 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880383132139-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880383132139-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880383132139-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880383132139-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36847
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880383132139-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:50:53.119587 18708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:50:53.119911 18708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:50:53.120028 18708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:50:53.125255 18708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:50:53.125346 18708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:50:53.125411 18708 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 14:50:53.131347 18708 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880383132139-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880383132139-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880383132139-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880383132139-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36847
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18708
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880383132139-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:50:53.132769 18708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:50:53.134262 18708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:50:53.145514 18713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:53.145473 18714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:53.145473 18716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:50:53.146283 18708 server_base.cc:1034] running on GCE node
I20260619 14:50:53.146833 18708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:50:53.147531 18708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:50:53.148764 18708 hybrid_clock.cc:648] HybridClock initialized: now 1781880653148743 us; error 40 us; skew 500 ppm
Jun 19 14:50:53 dist-test-slave-2sb1 krb5kdc[18467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880653, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:50:53.152165 18708 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 14:50:53.153450 18708 webserver.cc:460] Webserver started at http://127.0.98.2:33813/ using document root <none> and password file <none>
I20260619 14:50:53.154119 18708 fs_manager.cc:359] Metadata directory not provided
I20260619 14:50:53.154218 18708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:50:53.154472 18708 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:50:53.156873 18708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880383132139-392-0/minicluster-data/ts-1/data/instance:
uuid: "f0bb8dc9237342ddb0f031a8288ccd38"
format_stamp: "Formatted at 2026-06-19 14:50:53 on dist-test-slave-2sb1"
server_key: "cc42c2596020d459541b38c16e6bc7e2"
server_key_iv: "f1cad1fdd4d30c6dba446caee35d95b3"
server_key_version: "encryptionkey@0"
I20260619 14:50:53.157758 18708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880383132139-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f0bb8dc9237342ddb0f031a8288ccd38"
format_stamp: "Formatted at 2026-06-19 14:50:53 on dist-test-slave-2sb1"
server_key: "cc42c2596020d459541b38c16e6bc7e2"
server_key_iv: "f1cad1fdd4d30c6dba446caee35d95b3"
server_key_version: "encryptionkey@0"
I20260619 14:50:53.162343 18708 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 14:50:53.166193 18723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:53.167966 18708 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:50:53.168135 18708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880383132139-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880383132139-392-0/minicluster-data/ts-1/wal
uuid: "f0bb8dc9237342ddb0f031a8288ccd38"
format_stamp: "Formatted at 2026-06-19 14:50:53 on dist-test-slave-2sb1"
server_key: "cc42c2596020d459541b38c16e6bc7e2"
server_key_iv: "f1cad1fdd4d30c6dba446caee35d95b3"
server_key_version: "encryptionkey@0"
I20260619 14:50:53.168282 18708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880383132139-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880383132139-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880383132139-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:50:53.185961 18708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:50:53.189977 18708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:50:53.190222 18708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:50:53.190996 18708 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:50:53.191962 18708 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:50:53.192013 18708 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:53.192090 18708 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:50:53.192119 18708 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:53.202637 18708 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40283
I20260619 14:50:53.202670 18836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40283 every 8 connection(s)
I20260619 14:50:53.203676 18708 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880383132139-392-0/minicluster-data/ts-1/data/info.pb
I20260619 14:50:53.213765 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 18708
I20260619 14:50:53.213938 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880383132139-392-0/minicluster-data/ts-1/wal/instance
I20260619 14:50:53.214318 392 external_mini_cluster.cc:1428] Setting key e668e8734a0afe737e3112eb4441edc8
Jun 19 14:50:53 dist-test-slave-2sb1 krb5kdc[18467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880653, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:50:53.222030 18551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:53.206200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59245 (local address 127.0.98.62:36847)
0619 14:50:53.206488 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:53.206493 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:53.207918 (+ 1425us) server_negotiation.cc:407] Connection header received
0619 14:50:53.208906 (+ 988us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:53.208910 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:53.208962 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:53.209055 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:53.210759 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:53.211305 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:53.212855 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:53.213154 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:53.213259 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:53.216787 (+ 3528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:53.216795 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:53.216801 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:53.216838 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:53.218902 (+ 2064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:53.219706 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:53.219714 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:53.219719 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:53.219831 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:53.220312 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:53.220319 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:53.220324 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:53.220691 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:53.220815 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:53.221556 (+ 741us) server_negotiation.cc:299] Negotiation successful
0619 14:50:53.221734 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260619 14:50:53.222680 18839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:53.206768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36847 (local address 127.0.98.2:59245)
0619 14:50:53.207725 (+ 957us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:53.207773 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:53.208599 (+ 826us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:53.209338 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:53.209347 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:53.209993 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:53.210545 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:53.210563 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:53.211614 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:53.211617 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:53.212625 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:53.212637 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:53.213430 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:53.213434 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:53.213735 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:53.214574 (+ 839us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:53.214603 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:53.216521 (+ 1918us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:50:53.219064 (+ 2543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:53.219074 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:53.219099 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:53.219557 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:53.220021 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:53.220024 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:53.220025 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:53.220186 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:53.220820 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:50:53.220826 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:50:53.221237 (+ 411us) client_negotiation.cc:769] Sending connection context
0619 14:50:53.221444 (+ 207us) client_negotiation.cc:240] Negotiation successful
0619 14:50:53.221743 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":198,"threads_started":1}
I20260619 14:50:53.224205 18837 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36847
I20260619 14:50:53.224536 18837 heartbeater.cc:464] Registering TS with master...
I20260619 14:50:53.225282 18837 heartbeater.cc:511] Master 127.0.98.62:36847 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260619 14:50:53.226752 18512 ts_manager.cc:194] Registered new tserver with Master: f0bb8dc9237342ddb0f031a8288ccd38 (127.0.98.2:40283)
I20260619 14:50:53.227401 18512 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:59245
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880383132139-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880383132139-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 14:50:53.250285 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880383132139-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880383132139-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880383132139-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880383132139-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36847
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880383132139-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:50:53.386053 18842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:50:53.386386 18842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:50:53.386435 18842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:50:53.391072 18842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:50:53.391165 18842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:50:53.391229 18842 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 14:50:53.396987 18842 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880383132139-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880383132139-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880383132139-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880383132139-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36847
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18842
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880383132139-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:50:53.398249 18842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:50:53.399459 18842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:50:53.409891 18847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:53.410099 18848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:53.410127 18850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:50:53.410221 18842 server_base.cc:1034] running on GCE node
I20260619 14:50:53.411217 18842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:50:53.412086 18842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:50:53.413332 18842 hybrid_clock.cc:648] HybridClock initialized: now 1781880653413313 us; error 35 us; skew 500 ppm
Jun 19 14:50:53 dist-test-slave-2sb1 krb5kdc[18467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880653, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:50:53.416923 18842 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 14:50:53.418030 18842 webserver.cc:460] Webserver started at http://127.0.98.3:42909/ using document root <none> and password file <none>
I20260619 14:50:53.418761 18842 fs_manager.cc:359] Metadata directory not provided
I20260619 14:50:53.418867 18842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:50:53.419118 18842 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:50:53.421321 18842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880383132139-392-0/minicluster-data/ts-2/data/instance:
uuid: "8dc58bb3566142f08b84f21805e8ac28"
format_stamp: "Formatted at 2026-06-19 14:50:53 on dist-test-slave-2sb1"
server_key: "bf7789542e1029ffecf90eb69adaaa4e"
server_key_iv: "d78d9826bd1f88cb04183b75bfb81db8"
server_key_version: "encryptionkey@0"
I20260619 14:50:53.421943 18842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880383132139-392-0/minicluster-data/ts-2/wal/instance:
uuid: "8dc58bb3566142f08b84f21805e8ac28"
format_stamp: "Formatted at 2026-06-19 14:50:53 on dist-test-slave-2sb1"
server_key: "bf7789542e1029ffecf90eb69adaaa4e"
server_key_iv: "d78d9826bd1f88cb04183b75bfb81db8"
server_key_version: "encryptionkey@0"
I20260619 14:50:53.426067 18842 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260619 14:50:53.429188 18857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:53.430987 18842 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 14:50:53.431137 18842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880383132139-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880383132139-392-0/minicluster-data/ts-2/wal
uuid: "8dc58bb3566142f08b84f21805e8ac28"
format_stamp: "Formatted at 2026-06-19 14:50:53 on dist-test-slave-2sb1"
server_key: "bf7789542e1029ffecf90eb69adaaa4e"
server_key_iv: "d78d9826bd1f88cb04183b75bfb81db8"
server_key_version: "encryptionkey@0"
I20260619 14:50:53.431309 18842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880383132139-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880383132139-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880383132139-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:50:53.469908 18842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:50:53.473737 18842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:50:53.474004 18842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:50:53.474634 18842 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:50:53.475544 18842 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:50:53.475596 18842 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:53.475668 18842 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:50:53.475700 18842 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:53.485689 18842 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43675
I20260619 14:50:53.485767 18970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43675 every 8 connection(s)
I20260619 14:50:53.487069 18842 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880383132139-392-0/minicluster-data/ts-2/data/info.pb
I20260619 14:50:53.489626 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 18842
I20260619 14:50:53.489774 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880383132139-392-0/minicluster-data/ts-2/wal/instance
I20260619 14:50:53.490079 392 external_mini_cluster.cc:1428] Setting key 955da37e043a03d5c6d3249cb0f08064
Jun 19 14:50:53 dist-test-slave-2sb1 krb5kdc[18467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880653, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:50:53.504002 18551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:53.489377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57613 (local address 127.0.98.62:36847)
0619 14:50:53.489514 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:53.489517 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:53.490789 (+ 1272us) server_negotiation.cc:407] Connection header received
0619 14:50:53.492759 (+ 1970us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:53.492762 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:53.492834 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:53.492929 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:53.494317 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:53.494992 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:53.496213 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:53.496545 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:53.496640 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:53.499683 (+ 3043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:53.499686 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:53.499688 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:53.499723 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:53.501456 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:53.502117 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:53.502120 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:53.502122 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:53.502214 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:53.502646 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:53.502649 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:53.502651 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:53.502868 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:53.502966 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:53.503678 (+ 712us) server_negotiation.cc:299] Negotiation successful
0619 14:50:53.503816 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 14:50:53.505146 18973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:53.489896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36847 (local address 127.0.98.3:57613)
0619 14:50:53.490538 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:53.490626 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:53.492480 (+ 1854us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:53.493170 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:53.493178 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:53.493590 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:53.494146 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:53.494162 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:53.495141 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:53.495144 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:53.496079 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:53.496089 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:53.496761 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:53.496765 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:53.496959 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:53.497653 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:53.497670 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:53.499500 (+ 1830us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:50:53.501657 (+ 2157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:53.501663 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:53.501677 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:53.501986 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:53.502358 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:53.502360 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:53.502362 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:53.502514 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:53.503010 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:50:53.503016 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:50:53.503277 (+ 261us) client_negotiation.cc:769] Sending connection context
0619 14:50:53.503515 (+ 238us) client_negotiation.cc:240] Negotiation successful
0619 14:50:53.503877 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":160,"threads_started":1}
I20260619 14:50:53.506610 18971 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36847
I20260619 14:50:53.506889 18971 heartbeater.cc:464] Registering TS with master...
I20260619 14:50:53.507438 18971 heartbeater.cc:511] Master 127.0.98.62:36847 requested a full tablet report, sending...
I20260619 14:50:53.508795 18512 ts_manager.cc:194] Registered new tserver with Master: 8dc58bb3566142f08b84f21805e8ac28 (127.0.98.3:43675)
I20260619 14:50:53.509608 18512 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57613
I20260619 14:50:53.517944 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:50:53.529366 18551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:53.520243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36478 (local address 127.0.98.62:36847)
0619 14:50:53.520373 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:53.520377 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:53.520557 (+ 180us) server_negotiation.cc:407] Connection header received
0619 14:50:53.520644 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:53.520647 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:53.520738 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:53.520823 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:53.521865 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:53.522477 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:53.523546 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:53.523767 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:53.523864 (+ 97us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:53.525800 (+ 1936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:53.525803 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:53.525805 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:53.525831 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:53.527339 (+ 1508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:53.527834 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:53.527837 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:53.527838 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:53.527934 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:53.528274 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:53.528277 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:53.528279 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:53.528535 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:53.528687 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:53.529136 (+ 449us) server_negotiation.cc:299] Negotiation successful
0619 14:50:53.529231 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260619 14:50:53.533282 18512 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36478:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:50:53.536535 18512 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:50:53.553614 18983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:53.546986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40283 (local address 127.0.0.1:55204)
0619 14:50:53.547695 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:53.547747 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:53.547882 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:53.548694 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:53.548704 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:53.548732 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:50:53.548878 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:53.548886 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:53.550166 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:53.550170 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:53.551475 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:53.551488 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:53.552902 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:53.552905 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:53.553108 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:53.553164 (+ 56us) client_negotiation.cc:769] Sending connection context
0619 14:50:53.553293 (+ 129us) client_negotiation.cc:240] Negotiation successful
0619 14:50:53.553439 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"mutex_wait_us":69,"thread_start_us":132,"threads_started":1}
I20260619 14:50:53.553768 18984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:53.547196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55204 (local address 127.0.98.2:40283)
0619 14:50:53.547785 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:53.547800 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:53.547826 (+ 26us) server_negotiation.cc:407] Connection header received
0619 14:50:53.548078 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:53.548084 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:53.548302 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:53.548435 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:50:53.549035 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:53.550013 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:53.551813 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:53.552746 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:53.552870 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:53.553043 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:53.553461 (+ 418us) server_negotiation.cc:299] Negotiation successful
0619 14:50:53.553584 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":108,"threads_started":1}
I20260619 14:50:53.553974 18987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:53.548108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37428 (local address 127.0.98.3:43675)
0619 14:50:53.548423 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:53.548435 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:53.548923 (+ 488us) server_negotiation.cc:407] Connection header received
0619 14:50:53.549009 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:53.549014 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:53.549173 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:53.549297 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:50:53.549620 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:53.550727 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:53.552198 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:53.552802 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:53.552917 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:53.553179 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:53.553692 (+ 513us) server_negotiation.cc:299] Negotiation successful
0619 14:50:53.553807 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":98,"threads_started":1}
I20260619 14:50:53.554247 18982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:53.547758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43675 (local address 127.0.0.1:37428)
0619 14:50:53.548801 (+ 1043us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:53.548814 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:53.548910 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:53.549415 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:53.549417 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:53.549432 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:50:53.549503 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:53.549508 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:53.550861 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:53.550864 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:53.552056 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:53.552066 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:53.553255 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:53.553261 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:53.553408 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:53.553437 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 14:50:53.553505 (+ 68us) client_negotiation.cc:240] Negotiation successful
0619 14:50:53.553566 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":987,"mutex_wait_us":66,"thread_start_us":87,"threads_started":1}
I20260619 14:50:53.554546 18985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:53.547360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34386 (local address 127.0.98.1:43603)
0619 14:50:53.548236 (+ 876us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:53.548243 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:53.548693 (+ 450us) server_negotiation.cc:407] Connection header received
0619 14:50:53.548827 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:53.548839 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:53.549031 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:53.549185 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:50:53.549379 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:53.550562 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:53.552150 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:53.553053 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:53.553208 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:53.553353 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:53.553558 (+ 205us) server_negotiation.cc:299] Negotiation successful
0619 14:50:53.553651 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":755,"thread_start_us":100,"threads_started":1}
I20260619 14:50:53.554996 18986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:53.547186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43603 (local address 127.0.0.1:34386)
0619 14:50:53.548570 (+ 1384us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:53.548581 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:53.548711 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:53.549177 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:53.549179 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:53.549198 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:50:53.549281 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:53.549287 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:53.550695 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:53.550698 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:53.552006 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:53.552018 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:53.553207 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:53.553212 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:53.553325 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:53.553410 (+ 85us) client_negotiation.cc:769] Sending connection context
0619 14:50:53.554764 (+ 1354us) client_negotiation.cc:240] Negotiation successful
0619 14:50:53.554862 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1333,"mutex_wait_us":27,"thread_start_us":114,"threads_started":1}
I20260619 14:50:53.556993 18905 tablet_service.cc:1467] Processing CreateTablet for tablet 63699afad287465b868899820cd2326c (DEFAULT_TABLE table=test-table [id=f015d43e59b643ee82139efe4060c391]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:50:53.557125 18771 tablet_service.cc:1467] Processing CreateTablet for tablet 63699afad287465b868899820cd2326c (DEFAULT_TABLE table=test-table [id=f015d43e59b643ee82139efe4060c391]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:50:53.558122 18905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63699afad287465b868899820cd2326c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:50:53.558162 18771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63699afad287465b868899820cd2326c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:50:53.559824 18637 tablet_service.cc:1467] Processing CreateTablet for tablet 63699afad287465b868899820cd2326c (DEFAULT_TABLE table=test-table [id=f015d43e59b643ee82139efe4060c391]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:50:53.561434 18637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63699afad287465b868899820cd2326c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:50:53.564843 18988 tablet_bootstrap.cc:492] T 63699afad287465b868899820cd2326c P f0bb8dc9237342ddb0f031a8288ccd38: Bootstrap starting.
I20260619 14:50:53.566905 18989 tablet_bootstrap.cc:492] T 63699afad287465b868899820cd2326c P 8dc58bb3566142f08b84f21805e8ac28: Bootstrap starting.
I20260619 14:50:53.569065 18989 tablet_bootstrap.cc:654] T 63699afad287465b868899820cd2326c P 8dc58bb3566142f08b84f21805e8ac28: Neither blocks nor log segments found. Creating new log.
I20260619 14:50:53.569173 18988 tablet_bootstrap.cc:654] T 63699afad287465b868899820cd2326c P f0bb8dc9237342ddb0f031a8288ccd38: Neither blocks nor log segments found. Creating new log.
I20260619 14:50:53.570219 18989 log.cc:824] T 63699afad287465b868899820cd2326c P 8dc58bb3566142f08b84f21805e8ac28: Log is configured to *not* fsync() on all Append() calls
I20260619 14:50:53.570505 18988 log.cc:824] T 63699afad287465b868899820cd2326c P f0bb8dc9237342ddb0f031a8288ccd38: Log is configured to *not* fsync() on all Append() calls
I20260619 14:50:53.571237 18991 tablet_bootstrap.cc:492] T 63699afad287465b868899820cd2326c P 1630bb912ac143ef9bfa21bc3546e444: Bootstrap starting.
I20260619 14:50:53.572588 18989 tablet_bootstrap.cc:492] T 63699afad287465b868899820cd2326c P 8dc58bb3566142f08b84f21805e8ac28: No bootstrap required, opened a new log
I20260619 14:50:53.572835 18989 ts_tablet_manager.cc:1397] T 63699afad287465b868899820cd2326c P 8dc58bb3566142f08b84f21805e8ac28: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260619 14:50:53.574047 18991 tablet_bootstrap.cc:654] T 63699afad287465b868899820cd2326c P 1630bb912ac143ef9bfa21bc3546e444: Neither blocks nor log segments found. Creating new log.
I20260619 14:50:53.574149 18988 tablet_bootstrap.cc:492] T 63699afad287465b868899820cd2326c P f0bb8dc9237342ddb0f031a8288ccd38: No bootstrap required, opened a new log
I20260619 14:50:53.574350 18988 ts_tablet_manager.cc:1397] T 63699afad287465b868899820cd2326c P f0bb8dc9237342ddb0f031a8288ccd38: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20260619 14:50:53.574884 18991 log.cc:824] T 63699afad287465b868899820cd2326c P 1630bb912ac143ef9bfa21bc3546e444: Log is configured to *not* fsync() on all Append() calls
I20260619 14:50:53.576788 18991 tablet_bootstrap.cc:492] T 63699afad287465b868899820cd2326c P 1630bb912ac143ef9bfa21bc3546e444: No bootstrap required, opened a new log
I20260619 14:50:53.577013 18991 ts_tablet_manager.cc:1397] T 63699afad287465b868899820cd2326c P 1630bb912ac143ef9bfa21bc3546e444: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260619 14:50:53.577246 18989 raft_consensus.cc:348] T 63699afad287465b868899820cd2326c P 8dc58bb3566142f08b84f21805e8ac28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0bb8dc9237342ddb0f031a8288ccd38" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40283 } } peers { permanent_uuid: "8dc58bb3566142f08b84f21805e8ac28" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43675 } } peers { permanent_uuid: "1630bb912ac143ef9bfa21bc3546e444" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43603 } }
I20260619 14:50:53.577572 18989 raft_consensus.cc:374] T 63699afad287465b868899820cd2326c P 8dc58bb3566142f08b84f21805e8ac28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:50:53.577630 18989 raft_consensus.cc:729] T 63699afad287465b868899820cd2326c P 8dc58bb3566142f08b84f21805e8ac28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dc58bb3566142f08b84f21805e8ac28, State: Initialized, Role: FOLLOWER
I20260619 14:50:53.578153 18989 consensus_queue.cc:260] T 63699afad287465b868899820cd2326c P 8dc58bb3566142f08b84f21805e8ac28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0bb8dc9237342ddb0f031a8288ccd38" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40283 } } peers { permanent_uuid: "8dc58bb3566142f08b84f21805e8ac28" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43675 } } peers { permanent_uuid: "1630bb912ac143ef9bfa21bc3546e444" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43603 } }
I20260619 14:50:53.578337 18988 raft_consensus.cc:348] T 63699afad287465b868899820cd2326c P f0bb8dc9237342ddb0f031a8288ccd38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0bb8dc9237342ddb0f031a8288ccd38" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40283 } } peers { permanent_uuid: "8dc58bb3566142f08b84f21805e8ac28" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43675 } } peers { permanent_uuid: "1630bb912ac143ef9bfa21bc3546e444" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43603 } }
I20260619 14:50:53.578616 18988 raft_consensus.cc:374] T 63699afad287465b868899820cd2326c P f0bb8dc9237342ddb0f031a8288ccd38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:50:53.578676 18988 raft_consensus.cc:729] T 63699afad287465b868899820cd2326c P f0bb8dc9237342ddb0f031a8288ccd38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0bb8dc9237342ddb0f031a8288ccd38, State: Initialized, Role: FOLLOWER
I20260619 14:50:53.579140 18971 heartbeater.cc:503] Master 127.0.98.62:36847 was elected leader, sending a full tablet report...
I20260619 14:50:53.579277 18989 ts_tablet_manager.cc:1428] T 63699afad287465b868899820cd2326c P 8dc58bb3566142f08b84f21805e8ac28: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260619 14:50:53.580210 18988 consensus_queue.cc:260] T 63699afad287465b868899820cd2326c P f0bb8dc9237342ddb0f031a8288ccd38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0bb8dc9237342ddb0f031a8288ccd38" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40283 } } peers { permanent_uuid: "8dc58bb3566142f08b84f21805e8ac28" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43675 } } peers { permanent_uuid: "1630bb912ac143ef9bfa21bc3546e444" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43603 } }
I20260619 14:50:53.581818 18991 raft_consensus.cc:348] T 63699afad287465b868899820cd2326c P 1630bb912ac143ef9bfa21bc3546e444 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0bb8dc9237342ddb0f031a8288ccd38" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40283 } } peers { permanent_uuid: "8dc58bb3566142f08b84f21805e8ac28" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43675 } } peers { permanent_uuid: "1630bb912ac143ef9bfa21bc3546e444" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43603 } }
I20260619 14:50:53.582134 18991 raft_consensus.cc:374] T 63699afad287465b868899820cd2326c P 1630bb912ac143ef9bfa21bc3546e444 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:50:53.582204 18991 raft_consensus.cc:729] T 63699afad287465b868899820cd2326c P 1630bb912ac143ef9bfa21bc3546e444 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1630bb912ac143ef9bfa21bc3546e444, State: Initialized, Role: FOLLOWER
I20260619 14:50:53.582755 18991 consensus_queue.cc:260] T 63699afad287465b868899820cd2326c P 1630bb912ac143ef9bfa21bc3546e444 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0bb8dc9237342ddb0f031a8288ccd38" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40283 } } peers { permanent_uuid: "8dc58bb3566142f08b84f21805e8ac28" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43675 } } peers { permanent_uuid: "1630bb912ac143ef9bfa21bc3546e444" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43603 } }
I20260619 14:50:53.582937 18837 heartbeater.cc:503] Master 127.0.98.62:36847 was elected leader, sending a full tablet report...
I20260619 14:50:53.583294 18988 ts_tablet_manager.cc:1428] T 63699afad287465b868899820cd2326c P f0bb8dc9237342ddb0f031a8288ccd38: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20260619 14:50:53.583834 18991 ts_tablet_manager.cc:1428] T 63699afad287465b868899820cd2326c P 1630bb912ac143ef9bfa21bc3546e444: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260619 14:50:53.584744 18703 heartbeater.cc:503] Master 127.0.98.62:36847 was elected leader, sending a full tablet report...
W20260619 14:50:53.700918 18704 tablet.cc:2369] T 63699afad287465b868899820cd2326c P 1630bb912ac143ef9bfa21bc3546e444: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 14:50:53.705044 18838 tablet.cc:2369] T 63699afad287465b868899820cd2326c P f0bb8dc9237342ddb0f031a8288ccd38: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 14:50:53.738615 18972 tablet.cc:2369] T 63699afad287465b868899820cd2326c P 8dc58bb3566142f08b84f21805e8ac28: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:50:53.804733 18996 raft_consensus.cc:482] T 63699afad287465b868899820cd2326c P 1630bb912ac143ef9bfa21bc3546e444 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:50:53.805008 18996 raft_consensus.cc:504] T 63699afad287465b868899820cd2326c P 1630bb912ac143ef9bfa21bc3546e444 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0bb8dc9237342ddb0f031a8288ccd38" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40283 } } peers { permanent_uuid: "8dc58bb3566142f08b84f21805e8ac28" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43675 } } peers { permanent_uuid: "1630bb912ac143ef9bfa21bc3546e444" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43603 } }
I20260619 14:50:53.806397 18996 leader_election.cc:288] T 63699afad287465b868899820cd2326c P 1630bb912ac143ef9bfa21bc3546e444 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f0bb8dc9237342ddb0f031a8288ccd38 (127.0.98.2:40283), 8dc58bb3566142f08b84f21805e8ac28 (127.0.98.3:43675)
I20260619 14:50:53.809113 18994 raft_consensus.cc:482] T 63699afad287465b868899820cd2326c P 8dc58bb3566142f08b84f21805e8ac28 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:50:53.809324 18994 raft_consensus.cc:504] T 63699afad287465b868899820cd2326c P 8dc58bb3566142f08b84f21805e8ac28 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0bb8dc9237342ddb0f031a8288ccd38" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40283 } } peers { permanent_uuid: "8dc58bb3566142f08b84f21805e8ac28" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43675 } } peers { permanent_uuid: "1630bb912ac143ef9bfa21bc3546e444" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43603 } }
I20260619 14:50:53.810614 18994 leader_election.cc:288] T 63699afad287465b868899820cd2326c P 8dc58bb3566142f08b84f21805e8ac28 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f0bb8dc9237342ddb0f031a8288ccd38 (127.0.98.2:40283), 1630bb912ac143ef9bfa21bc3546e444 (127.0.98.1:43603)
I20260619 14:50:53.812359 18984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:53.806942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55945 (local address 127.0.98.2:40283)
0619 14:50:53.807181 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:53.807186 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:53.807337 (+ 151us) server_negotiation.cc:407] Connection header received
0619 14:50:53.807615 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:53.807618 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:53.807663 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:53.807743 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:50:53.808122 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:53.808753 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:53.811106 (+ 2353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:53.811717 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:53.811793 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:53.811870 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:53.812165 (+ 295us) server_negotiation.cc:299] Negotiation successful
0619 14:50:53.812237 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134}
I20260619 14:50:53.812388 18997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:53.806746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40283 (local address 127.0.98.1:55945)
0619 14:50:53.807202 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:53.807218 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:53.807385 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:53.807883 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:53.807885 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:53.807900 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:50:53.808009 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:53.808014 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:53.808968 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:53.808971 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:53.810707 (+ 1736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:53.810722 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:53.811887 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:53.811893 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:53.812025 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:53.812054 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 14:50:53.812145 (+ 91us) client_negotiation.cc:240] Negotiation successful
0619 14:50:53.812223 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"mutex_wait_us":42,"thread_start_us":150,"threads_started":1}
I20260619 14:50:53.813021 18791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63699afad287465b868899820cd2326c" candidate_uuid: "1630bb912ac143ef9bfa21bc3546e444" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0bb8dc9237342ddb0f031a8288ccd38" is_pre_election: true
I20260619 14:50:53.813359 18791 raft_consensus.cc:2454] T 63699afad287465b868899820cd2326c P f0bb8dc9237342ddb0f031a8288ccd38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1630bb912ac143ef9bfa21bc3546e444 in term 0.
I20260619 14:50:53.813948 18987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:53.807395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60661 (local address 127.0.98.3:43675)
0619 14:50:53.807531 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:53.807536 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:53.807792 (+ 256us) server_negotiation.cc:407] Connection header received
0619 14:50:53.808046 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:53.808049 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:53.808092 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:53.808181 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:50:53.808462 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:53.809162 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:53.810837 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:53.811437 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:53.811510 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:53.811571 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:53.813727 (+ 2156us) server_negotiation.cc:299] Negotiation successful
0619 14:50:53.813823 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":60}
I20260619 14:50:53.813903 18590 leader_election.cc:302] T 63699afad287465b868899820cd2326c P 1630bb912ac143ef9bfa21bc3546e444 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1630bb912ac143ef9bfa21bc3546e444, f0bb8dc9237342ddb0f031a8288ccd38; no voters:
I20260619 14:50:53.814383 18998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:53.807246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43675 (local address 127.0.98.1:60661)
0619 14:50:53.807642 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:53.807681 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:53.807833 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:53.808202 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:53.808205 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:53.808221 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:50:53.808338 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:53.808343 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:53.809341 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:53.809344 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:53.810669 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:53.810682 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:53.811994 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:53.811998 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:53.812111 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:53.812132 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 14:50:53.814133 (+ 2001us) client_negotiation.cc:240] Negotiation successful
0619 14:50:53.814230 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"mutex_wait_us":40,"thread_start_us":108,"threads_started":1}
I20260619 14:50:53.814383 18996 raft_consensus.cc:2781] T 63699afad287465b868899820cd2326c P 1630bb912ac143ef9bfa21bc3546e444 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:50:53.814491 18996 raft_consensus.cc:482] T 63699afad287465b868899820cd2326c P 1630bb912ac143ef9bfa21bc3546e444 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:50:53.814553 18996 raft_consensus.cc:3037] T 63699afad287465b868899820cd2326c P 1630bb912ac143ef9bfa21bc3546e444 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:50:53.815423 18973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:53.810797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40283 (local address 127.0.98.3:55589)
0619 14:50:53.810920 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:53.810935 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:53.811011 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:53.811732 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:53.811735 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:53.811749 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:50:53.811861 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:53.811867 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:53.812856 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:53.812860 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:53.814078 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:53.814088 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:53.814991 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:53.814995 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:53.815094 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:53.815115 (+ 21us) client_negotiation.cc:769] Sending connection context
0619 14:50:53.815222 (+ 107us) client_negotiation.cc:240] Negotiation successful
0619 14:50:53.815301 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260619 14:50:53.815464 19000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:53.811075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55589 (local address 127.0.98.2:40283)
0619 14:50:53.811486 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:53.811490 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:53.811505 (+ 15us) server_negotiation.cc:407] Connection header received
0619 14:50:53.811554 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:53.811556 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:53.811601 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:53.811718 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:50:53.811949 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:53.812734 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:53.814230 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:53.814852 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:53.814989 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:53.815048 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:53.815239 (+ 191us) server_negotiation.cc:299] Negotiation successful
0619 14:50:53.815312 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":58,"threads_started":1}
I20260619 14:50:53.815950 18791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63699afad287465b868899820cd2326c" candidate_uuid: "8dc58bb3566142f08b84f21805e8ac28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0bb8dc9237342ddb0f031a8288ccd38" is_pre_election: true
I20260619 14:50:53.816025 18985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:53.811308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57145 (local address 127.0.98.1:43603)
0619 14:50:53.811670 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:53.811675 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:53.811686 (+ 11us) server_negotiation.cc:407] Connection header received
0619 14:50:53.811728 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:53.811731 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:53.811781 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:53.811850 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:50:53.812586 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:53.813307 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:53.814707 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:53.815293 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:53.815404 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:53.815459 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:53.815806 (+ 347us) server_negotiation.cc:299] Negotiation successful
0619 14:50:53.815910 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287}
I20260619 14:50:53.816087 18791 raft_consensus.cc:2454] T 63699afad287465b868899820cd2326c P f0bb8dc9237342ddb0f031a8288ccd38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8dc58bb3566142f08b84f21805e8ac28 in term 0.
I20260619 14:50:53.816149 18999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:53.810899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43603 (local address 127.0.98.3:57145)
0619 14:50:53.811472 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:53.811486 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:53.811588 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:53.812236 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:53.812239 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:53.812257 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:50:53.812362 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:53.812367 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:53.813447 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:53.813450 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:53.814578 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:53.814587 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:53.815541 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:53.815545 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:53.815661 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:53.815688 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 14:50:53.815768 (+ 80us) client_negotiation.cc:240] Negotiation successful
0619 14:50:53.815846 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":159,"threads_started":1}
I20260619 14:50:53.816810 18858 leader_election.cc:302] T 63699afad287465b868899820cd2326c P 8dc58bb3566142f08b84f21805e8ac28 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8dc58bb3566142f08b84f21805e8ac28, f0bb8dc9237342ddb0f031a8288ccd38; no voters:
I20260619 14:50:53.816996 18657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63699afad287465b868899820cd2326c" candidate_uuid: "8dc58bb3566142f08b84f21805e8ac28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1630bb912ac143ef9bfa21bc3546e444" is_pre_election: true
I20260619 14:50:53.817091 18925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63699afad287465b868899820cd2326c" candidate_uuid: "1630bb912ac143ef9bfa21bc3546e444" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dc58bb3566142f08b84f21805e8ac28" is_pre_election: true
I20260619 14:50:53.817152 18994 raft_consensus.cc:2781] T 63699afad287465b868899820cd2326c P 8dc58bb3566142f08b84f21805e8ac28 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:50:53.817286 18925 raft_consensus.cc:2454] T 63699afad287465b868899820cd2326c P 8dc58bb3566142f08b84f21805e8ac28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1630bb912ac143ef9bfa21bc3546e444 in term 0.
I20260619 14:50:53.817353 18996 raft_consensus.cc:504] T 63699afad287465b868899820cd2326c P 1630bb912ac143ef9bfa21bc3546e444 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0bb8dc9237342ddb0f031a8288ccd38" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40283 } } peers { permanent_uuid: "8dc58bb3566142f08b84f21805e8ac28" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43675 } } peers { permanent_uuid: "1630bb912ac143ef9bfa21bc3546e444" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43603 } }
I20260619 14:50:53.817503 18994 raft_consensus.cc:482] T 63699afad287465b868899820cd2326c P 8dc58bb3566142f08b84f21805e8ac28 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:50:53.817569 18994 raft_consensus.cc:3037] T 63699afad287465b868899820cd2326c P 8dc58bb3566142f08b84f21805e8ac28 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:50:53.817683 18657 raft_consensus.cc:2379] T 63699afad287465b868899820cd2326c P 1630bb912ac143ef9bfa21bc3546e444 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8dc58bb3566142f08b84f21805e8ac28 in current term 1: Already voted for candidate 1630bb912ac143ef9bfa21bc3546e444 in this term.
I20260619 14:50:53.817984 18996 leader_election.cc:288] T 63699afad287465b868899820cd2326c P 1630bb912ac143ef9bfa21bc3546e444 [CANDIDATE]: Term 1 election: Requested vote from peers f0bb8dc9237342ddb0f031a8288ccd38 (127.0.98.2:40283), 8dc58bb3566142f08b84f21805e8ac28 (127.0.98.3:43675)
I20260619 14:50:53.818624 18925 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63699afad287465b868899820cd2326c" candidate_uuid: "1630bb912ac143ef9bfa21bc3546e444" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dc58bb3566142f08b84f21805e8ac28"
I20260619 14:50:53.818625 18791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63699afad287465b868899820cd2326c" candidate_uuid: "1630bb912ac143ef9bfa21bc3546e444" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0bb8dc9237342ddb0f031a8288ccd38"
I20260619 14:50:53.818756 18791 raft_consensus.cc:3037] T 63699afad287465b868899820cd2326c P f0bb8dc9237342ddb0f031a8288ccd38 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:50:53.818976 18994 raft_consensus.cc:504] T 63699afad287465b868899820cd2326c P 8dc58bb3566142f08b84f21805e8ac28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0bb8dc9237342ddb0f031a8288ccd38" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40283 } } peers { permanent_uuid: "8dc58bb3566142f08b84f21805e8ac28" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43675 } } peers { permanent_uuid: "1630bb912ac143ef9bfa21bc3546e444" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43603 } }
I20260619 14:50:53.819258 18925 raft_consensus.cc:2379] T 63699afad287465b868899820cd2326c P 8dc58bb3566142f08b84f21805e8ac28 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 1630bb912ac143ef9bfa21bc3546e444 in current term 1: Already voted for candidate 8dc58bb3566142f08b84f21805e8ac28 in this term.
I20260619 14:50:53.819383 18994 leader_election.cc:288] T 63699afad287465b868899820cd2326c P 8dc58bb3566142f08b84f21805e8ac28 [CANDIDATE]: Term 1 election: Requested vote from peers f0bb8dc9237342ddb0f031a8288ccd38 (127.0.98.2:40283), 1630bb912ac143ef9bfa21bc3546e444 (127.0.98.1:43603)
I20260619 14:50:53.819828 18790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63699afad287465b868899820cd2326c" candidate_uuid: "8dc58bb3566142f08b84f21805e8ac28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0bb8dc9237342ddb0f031a8288ccd38"
I20260619 14:50:53.819832 18657 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63699afad287465b868899820cd2326c" candidate_uuid: "8dc58bb3566142f08b84f21805e8ac28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1630bb912ac143ef9bfa21bc3546e444"
I20260619 14:50:53.820052 18657 raft_consensus.cc:2379] T 63699afad287465b868899820cd2326c P 1630bb912ac143ef9bfa21bc3546e444 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 8dc58bb3566142f08b84f21805e8ac28 in current term 1: Already voted for candidate 1630bb912ac143ef9bfa21bc3546e444 in this term.
I20260619 14:50:53.820330 18791 raft_consensus.cc:2454] T 63699afad287465b868899820cd2326c P f0bb8dc9237342ddb0f031a8288ccd38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1630bb912ac143ef9bfa21bc3546e444 in term 1.
I20260619 14:50:53.820602 18590 leader_election.cc:302] T 63699afad287465b868899820cd2326c P 1630bb912ac143ef9bfa21bc3546e444 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1630bb912ac143ef9bfa21bc3546e444, f0bb8dc9237342ddb0f031a8288ccd38; no voters: 8dc58bb3566142f08b84f21805e8ac28
I20260619 14:50:53.820977 18861 leader_election.cc:302] T 63699afad287465b868899820cd2326c P 8dc58bb3566142f08b84f21805e8ac28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8dc58bb3566142f08b84f21805e8ac28; no voters: 1630bb912ac143ef9bfa21bc3546e444, f0bb8dc9237342ddb0f031a8288ccd38
I20260619 14:50:53.821183 18994 raft_consensus.cc:2726] T 63699afad287465b868899820cd2326c P 8dc58bb3566142f08b84f21805e8ac28 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260619 14:50:53.821277 18996 raft_consensus.cc:2781] T 63699afad287465b868899820cd2326c P 1630bb912ac143ef9bfa21bc3546e444 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:50:53.821506 18996 raft_consensus.cc:686] T 63699afad287465b868899820cd2326c P 1630bb912ac143ef9bfa21bc3546e444 [term 1 LEADER]: Becoming Leader. State: Replica: 1630bb912ac143ef9bfa21bc3546e444, State: Running, Role: LEADER
I20260619 14:50:53.821877 18996 consensus_queue.cc:237] T 63699afad287465b868899820cd2326c P 1630bb912ac143ef9bfa21bc3546e444 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0bb8dc9237342ddb0f031a8288ccd38" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40283 } } peers { permanent_uuid: "8dc58bb3566142f08b84f21805e8ac28" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43675 } } peers { permanent_uuid: "1630bb912ac143ef9bfa21bc3546e444" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43603 } }
I20260619 14:50:53.825371 18511 catalog_manager.cc:5495] T 63699afad287465b868899820cd2326c P 1630bb912ac143ef9bfa21bc3546e444 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1630bb912ac143ef9bfa21bc3546e444 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "1630bb912ac143ef9bfa21bc3546e444" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0bb8dc9237342ddb0f031a8288ccd38" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40283 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8dc58bb3566142f08b84f21805e8ac28" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1630bb912ac143ef9bfa21bc3546e444" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43603 } health_report { overall_health: HEALTHY } } }
I20260619 14:50:53.858815 18985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:53.854086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34398 (local address 127.0.98.1:43603)
0619 14:50:53.854312 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:53.854316 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:53.854329 (+ 13us) server_negotiation.cc:407] Connection header received
0619 14:50:53.854413 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:53.854416 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:53.854492 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:53.854596 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:50:53.855038 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:53.855655 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:53.856727 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:53.857025 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:53.857124 (+ 99us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:53.857650 (+ 526us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:50:53.858203 (+ 553us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:50:53.858373 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:53.858643 (+ 270us) server_negotiation.cc:299] Negotiation successful
0619 14:50:53.858688 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146}
I20260619 14:50:53.865597 18791 raft_consensus.cc:1261] T 63699afad287465b868899820cd2326c P f0bb8dc9237342ddb0f031a8288ccd38 [term 1 FOLLOWER]: Refusing update from remote peer 1630bb912ac143ef9bfa21bc3546e444: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:50:53.865691 18925 raft_consensus.cc:1261] T 63699afad287465b868899820cd2326c P 8dc58bb3566142f08b84f21805e8ac28 [term 1 FOLLOWER]: Refusing update from remote peer 1630bb912ac143ef9bfa21bc3546e444: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:50:53.866566 19001 consensus_queue.cc:1035] T 63699afad287465b868899820cd2326c P 1630bb912ac143ef9bfa21bc3546e444 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0bb8dc9237342ddb0f031a8288ccd38" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40283 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:50:53.866822 18996 consensus_queue.cc:1035] T 63699afad287465b868899820cd2326c P 1630bb912ac143ef9bfa21bc3546e444 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dc58bb3566142f08b84f21805e8ac28" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:50:53.887130 18512 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36478:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:50:53.887344 18512 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36478:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:50:53.890333 18512 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b94d059e3a1844c59e878b66a2f18ca2: Sending DeleteTablet for 3 replicas of tablet 63699afad287465b868899820cd2326c
I20260619 14:50:53.891335 18905 tablet_service.cc:1514] Processing DeleteTablet for tablet 63699afad287465b868899820cd2326c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:50:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37428
I20260619 14:50:53.891283 18637 tablet_service.cc:1514] Processing DeleteTablet for tablet 63699afad287465b868899820cd2326c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:50:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34386
I20260619 14:50:53.891938 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 18574
I20260619 14:50:53.891999 19014 tablet_replica.cc:330] T 63699afad287465b868899820cd2326c P 8dc58bb3566142f08b84f21805e8ac28: stopping tablet replica
I20260619 14:50:53.892761 18771 tablet_service.cc:1514] Processing DeleteTablet for tablet 63699afad287465b868899820cd2326c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:50:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55204
I20260619 14:50:53.892808 19014 raft_consensus.cc:2229] T 63699afad287465b868899820cd2326c P 8dc58bb3566142f08b84f21805e8ac28 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:50:53.893188 19014 raft_consensus.cc:2258] T 63699afad287465b868899820cd2326c P 8dc58bb3566142f08b84f21805e8ac28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:50:53.893219 19016 tablet_replica.cc:330] T 63699afad287465b868899820cd2326c P f0bb8dc9237342ddb0f031a8288ccd38: stopping tablet replica
I20260619 14:50:53.893533 19016 raft_consensus.cc:2229] T 63699afad287465b868899820cd2326c P f0bb8dc9237342ddb0f031a8288ccd38 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:50:53.893821 19016 raft_consensus.cc:2258] T 63699afad287465b868899820cd2326c P f0bb8dc9237342ddb0f031a8288ccd38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:50:53.894954 19014 ts_tablet_manager.cc:1905] T 63699afad287465b868899820cd2326c P 8dc58bb3566142f08b84f21805e8ac28: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:50:53.895205 19016 ts_tablet_manager.cc:1905] T 63699afad287465b868899820cd2326c P f0bb8dc9237342ddb0f031a8288ccd38: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:50:53.897557 19014 ts_tablet_manager.cc:1918] T 63699afad287465b868899820cd2326c P 8dc58bb3566142f08b84f21805e8ac28: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:50:53.897688 19014 log.cc:1196] T 63699afad287465b868899820cd2326c P 8dc58bb3566142f08b84f21805e8ac28: Deleting WAL directory at /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880383132139-392-0/minicluster-data/ts-2/wal/wals/63699afad287465b868899820cd2326c
I20260619 14:50:53.898051 19014 ts_tablet_manager.cc:1939] T 63699afad287465b868899820cd2326c P 8dc58bb3566142f08b84f21805e8ac28: Deleting consensus metadata
I20260619 14:50:53.898797 19016 ts_tablet_manager.cc:1918] T 63699afad287465b868899820cd2326c P f0bb8dc9237342ddb0f031a8288ccd38: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:50:53.898918 19016 log.cc:1196] T 63699afad287465b868899820cd2326c P f0bb8dc9237342ddb0f031a8288ccd38: Deleting WAL directory at /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781880383132139-392-0/minicluster-data/ts-1/wal/wals/63699afad287465b868899820cd2326c
I20260619 14:50:53.899248 19016 ts_tablet_manager.cc:1939] T 63699afad287465b868899820cd2326c P f0bb8dc9237342ddb0f031a8288ccd38: Deleting consensus metadata
I20260619 14:50:53.899420 18496 catalog_manager.cc:4841] TS 8dc58bb3566142f08b84f21805e8ac28 (127.0.98.3:43675): tablet 63699afad287465b868899820cd2326c (table test-table [id=f015d43e59b643ee82139efe4060c391]) successfully deleted
I20260619 14:50:53.900606 18496 catalog_manager.cc:4841] TS f0bb8dc9237342ddb0f031a8288ccd38 (127.0.98.2:40283): tablet 63699afad287465b868899820cd2326c (table test-table [id=f015d43e59b643ee82139efe4060c391]) successfully deleted
W20260619 14:50:53.901233 18499 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43603 (error 108)
I20260619 14:50:53.902077 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 18708
I20260619 14:50:53.902501 18982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:53.902061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43603 (local address 127.0.0.1:34400)
0619 14:50:53.902315 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:53.902398 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150}
W20260619 14:50:53.902984 18499 catalog_manager.cc:4584] TS 1630bb912ac143ef9bfa21bc3546e444 (127.0.98.1:43603): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 63699afad287465b868899820cd2326c: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43603: connect: Connection refused (error 111)
I20260619 14:50:53.909050 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 18842
I20260619 14:50:53.915340 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 18480
2026-06-19T14:50:53Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2730 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-19T14:50:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:50:53Z Disabled control of system clock
I20260619 14:50:53.948886 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880383132139-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880383132139-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880383132139-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45481
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:40849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45481
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 14:50:54.075133 19021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:50:54.075508 19021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:50:54.075598 19021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:50:54.080446 19021 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:50:54.080706 19021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:50:54.080749 19021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:50:54.080811 19021 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:50:54.080868 19021 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:50:54.086982 19021 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880383132139-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880383132139-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45481
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19021
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880383132139-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:50:54.088232 19021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:50:54.089826 19021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:50:54.099489 19026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:54.099460 19027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:54.099463 19029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:50:54.099823 19021 server_base.cc:1034] running on GCE node
I20260619 14:50:54.100849 19021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:50:54.102171 19021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:50:54.103371 19021 hybrid_clock.cc:648] HybridClock initialized: now 1781880654103347 us; error 49 us; skew 500 ppm
I20260619 14:50:54.106101 19021 webserver.cc:460] Webserver started at http://127.0.98.62:34971/ using document root <none> and password file <none>
I20260619 14:50:54.106868 19021 fs_manager.cc:359] Metadata directory not provided
I20260619 14:50:54.106940 19021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:50:54.107133 19021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:50:54.109477 19021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880383132139-392-0/minicluster-data/master-0/data/instance:
uuid: "df2790574e5c4d41a30d61fbea6777e9"
format_stamp: "Formatted at 2026-06-19 14:50:54 on dist-test-slave-2sb1"
server_key: "28a448baf81c8325d77de14c806fac66"
server_key_iv: "5d442314296d7d3e35d1d72b144f5e59"
server_key_version: "encryptionkey@0"
I20260619 14:50:54.110061 19021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880383132139-392-0/minicluster-data/master-0/wal/instance:
uuid: "df2790574e5c4d41a30d61fbea6777e9"
format_stamp: "Formatted at 2026-06-19 14:50:54 on dist-test-slave-2sb1"
server_key: "28a448baf81c8325d77de14c806fac66"
server_key_iv: "5d442314296d7d3e35d1d72b144f5e59"
server_key_version: "encryptionkey@0"
I20260619 14:50:54.114555 19021 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 14:50:54.117574 19035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:54.119421 19021 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 14:50:54.119555 19021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880383132139-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880383132139-392-0/minicluster-data/master-0/wal
uuid: "df2790574e5c4d41a30d61fbea6777e9"
format_stamp: "Formatted at 2026-06-19 14:50:54 on dist-test-slave-2sb1"
server_key: "28a448baf81c8325d77de14c806fac66"
server_key_iv: "5d442314296d7d3e35d1d72b144f5e59"
server_key_version: "encryptionkey@0"
I20260619 14:50:54.119663 19021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880383132139-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880383132139-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880383132139-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:50:54.143152 19021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:50:54.143918 19021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:50:54.144110 19021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:50:54.152269 19021 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45481
I20260619 14:50:54.152307 19087 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45481 every 8 connection(s)
I20260619 14:50:54.153419 19021 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:50:54.156689 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 19021
I20260619 14:50:54.156849 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880383132139-392-0/minicluster-data/master-0/wal/instance
I20260619 14:50:54.157080 19088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:50:54.157171 392 external_mini_cluster.cc:1428] Setting key 028e6290d236a90ffd57cb66aa45864c
I20260619 14:50:54.164775 19088 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df2790574e5c4d41a30d61fbea6777e9: Bootstrap starting.
I20260619 14:50:54.167317 19088 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df2790574e5c4d41a30d61fbea6777e9: Neither blocks nor log segments found. Creating new log.
I20260619 14:50:54.167968 19091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:54.159100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57370 (local address 127.0.98.62:45481)
0619 14:50:54.159772 (+ 672us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:54.159789 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:54.159827 (+ 38us) server_negotiation.cc:407] Connection header received
0619 14:50:54.160548 (+ 721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:54.160568 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:54.161097 (+ 529us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:54.161421 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:54.161892 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:54.162920 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:54.164226 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:54.164944 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:54.165153 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:54.166121 (+ 968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:54.166126 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:54.166140 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:50:54.166554 (+ 414us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:54.166615 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:50:54.166635 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:54.166753 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:54.166878 (+ 125us) server_negotiation.cc:299] Negotiation successful
0619 14:50:54.167158 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":152,"threads_started":1}
I20260619 14:50:54.168185 19088 log.cc:824] T 00000000000000000000000000000000 P df2790574e5c4d41a30d61fbea6777e9: Log is configured to *not* fsync() on all Append() calls
I20260619 14:50:54.170547 19088 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df2790574e5c4d41a30d61fbea6777e9: No bootstrap required, opened a new log
I20260619 14:50:54.175294 19088 raft_consensus.cc:348] T 00000000000000000000000000000000 P df2790574e5c4d41a30d61fbea6777e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df2790574e5c4d41a30d61fbea6777e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45481 } }
I20260619 14:50:54.175618 19088 raft_consensus.cc:374] T 00000000000000000000000000000000 P df2790574e5c4d41a30d61fbea6777e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:50:54.175670 19088 raft_consensus.cc:729] T 00000000000000000000000000000000 P df2790574e5c4d41a30d61fbea6777e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df2790574e5c4d41a30d61fbea6777e9, State: Initialized, Role: FOLLOWER
I20260619 14:50:54.176258 19088 consensus_queue.cc:260] T 00000000000000000000000000000000 P df2790574e5c4d41a30d61fbea6777e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df2790574e5c4d41a30d61fbea6777e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45481 } }
I20260619 14:50:54.176412 19088 raft_consensus.cc:388] T 00000000000000000000000000000000 P df2790574e5c4d41a30d61fbea6777e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:50:54.176512 19088 raft_consensus.cc:482] T 00000000000000000000000000000000 P df2790574e5c4d41a30d61fbea6777e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:50:54.176692 19088 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df2790574e5c4d41a30d61fbea6777e9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:50:54.177923 19088 raft_consensus.cc:504] T 00000000000000000000000000000000 P df2790574e5c4d41a30d61fbea6777e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df2790574e5c4d41a30d61fbea6777e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45481 } }
I20260619 14:50:54.178380 19088 leader_election.cc:302] T 00000000000000000000000000000000 P df2790574e5c4d41a30d61fbea6777e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df2790574e5c4d41a30d61fbea6777e9; no voters:
I20260619 14:50:54.178781 19088 leader_election.cc:288] T 00000000000000000000000000000000 P df2790574e5c4d41a30d61fbea6777e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:50:54.179054 19093 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df2790574e5c4d41a30d61fbea6777e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:50:54.179355 19093 raft_consensus.cc:686] T 00000000000000000000000000000000 P df2790574e5c4d41a30d61fbea6777e9 [term 1 LEADER]: Becoming Leader. State: Replica: df2790574e5c4d41a30d61fbea6777e9, State: Running, Role: LEADER
I20260619 14:50:54.179853 19093 consensus_queue.cc:237] T 00000000000000000000000000000000 P df2790574e5c4d41a30d61fbea6777e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df2790574e5c4d41a30d61fbea6777e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45481 } }
I20260619 14:50:54.180338 19088 sys_catalog.cc:564] T 00000000000000000000000000000000 P df2790574e5c4d41a30d61fbea6777e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:50:54.181949 19095 sys_catalog.cc:455] T 00000000000000000000000000000000 P df2790574e5c4d41a30d61fbea6777e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader df2790574e5c4d41a30d61fbea6777e9. Latest consensus state: current_term: 1 leader_uuid: "df2790574e5c4d41a30d61fbea6777e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df2790574e5c4d41a30d61fbea6777e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45481 } } }
I20260619 14:50:54.182099 19095 sys_catalog.cc:458] T 00000000000000000000000000000000 P df2790574e5c4d41a30d61fbea6777e9 [sys.catalog]: This master's current role is: LEADER
I20260619 14:50:54.181912 19094 sys_catalog.cc:455] T 00000000000000000000000000000000 P df2790574e5c4d41a30d61fbea6777e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "df2790574e5c4d41a30d61fbea6777e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df2790574e5c4d41a30d61fbea6777e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45481 } } }
I20260619 14:50:54.182389 19094 sys_catalog.cc:458] T 00000000000000000000000000000000 P df2790574e5c4d41a30d61fbea6777e9 [sys.catalog]: This master's current role is: LEADER
I20260619 14:50:54.182619 19102 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:50:54.186026 19102 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:50:54.193029 19102 catalog_manager.cc:1347] Generated new cluster ID: 260f1496cbbd4692b14c09bd3a636ac9
I20260619 14:50:54.193137 19102 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:50:54.210610 19102 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:50:54.211704 19102 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:50:54.223088 19102 catalog_manager.cc:5868] T 00000000000000000000000000000000 P df2790574e5c4d41a30d61fbea6777e9: Generated new TSK 0
I20260619 14:50:54.224082 19102 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:50:54.236531 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880383132139-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880383132139-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880383132139-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880383132139-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45481
--builtin_ntp_servers=127.0.98.20:40849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 14:50:54.359223 19112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:50:54.359607 19112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:50:54.359690 19112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:50:54.364118 19112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:50:54.364248 19112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:50:54.364354 19112 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 14:50:54.370028 19112 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880383132139-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880383132139-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880383132139-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45481
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19112
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880383132139-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:50:54.371318 19112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:50:54.372917 19112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:50:54.383687 19117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:54.383687 19118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:54.383680 19120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:50:54.384030 19112 server_base.cc:1034] running on GCE node
I20260619 14:50:54.384452 19112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:50:54.385337 19112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:50:54.386848 19112 hybrid_clock.cc:648] HybridClock initialized: now 1781880654386805 us; error 57 us; skew 500 ppm
I20260619 14:50:54.389331 19112 webserver.cc:460] Webserver started at http://127.0.98.1:43239/ using document root <none> and password file <none>
I20260619 14:50:54.390040 19112 fs_manager.cc:359] Metadata directory not provided
I20260619 14:50:54.390153 19112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:50:54.390422 19112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:50:54.392843 19112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880383132139-392-0/minicluster-data/ts-0/data/instance:
uuid: "1b1f27fbeaf54820b618993e0daac38c"
format_stamp: "Formatted at 2026-06-19 14:50:54 on dist-test-slave-2sb1"
server_key: "9b25797fc044e9d4c1922643cc92c9db"
server_key_iv: "f65fb75a267beed1128946463226a1ef"
server_key_version: "encryptionkey@0"
I20260619 14:50:54.393585 19112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880383132139-392-0/minicluster-data/ts-0/wal/instance:
uuid: "1b1f27fbeaf54820b618993e0daac38c"
format_stamp: "Formatted at 2026-06-19 14:50:54 on dist-test-slave-2sb1"
server_key: "9b25797fc044e9d4c1922643cc92c9db"
server_key_iv: "f65fb75a267beed1128946463226a1ef"
server_key_version: "encryptionkey@0"
I20260619 14:50:54.398689 19112 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260619 14:50:54.401834 19126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:54.403689 19112 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:50:54.403858 19112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880383132139-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880383132139-392-0/minicluster-data/ts-0/wal
uuid: "1b1f27fbeaf54820b618993e0daac38c"
format_stamp: "Formatted at 2026-06-19 14:50:54 on dist-test-slave-2sb1"
server_key: "9b25797fc044e9d4c1922643cc92c9db"
server_key_iv: "f65fb75a267beed1128946463226a1ef"
server_key_version: "encryptionkey@0"
I20260619 14:50:54.404033 19112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880383132139-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880383132139-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880383132139-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:50:54.432365 19112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:50:54.433266 19112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:50:54.433498 19112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:50:54.434247 19112 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:50:54.435278 19112 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:50:54.435351 19112 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:54.435431 19112 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:50:54.435480 19112 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:54.445462 19112 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37231
I20260619 14:50:54.445515 19239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37231 every 8 connection(s)
I20260619 14:50:54.446496 19112 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880383132139-392-0/minicluster-data/ts-0/data/info.pb
I20260619 14:50:54.454434 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 19112
I20260619 14:50:54.454581 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880383132139-392-0/minicluster-data/ts-0/wal/instance
I20260619 14:50:54.454897 392 external_mini_cluster.cc:1428] Setting key b10f5355ea6ec3feebb80c69e6b8e3f1
I20260619 14:50:54.457911 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880383132139-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880383132139-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880383132139-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880383132139-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45481
--builtin_ntp_servers=127.0.98.20:40849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 14:50:54.459297 19091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:54.448577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46005 (local address 127.0.98.62:45481)
0619 14:50:54.448783 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:54.448787 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:54.450213 (+ 1426us) server_negotiation.cc:407] Connection header received
0619 14:50:54.451719 (+ 1506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:54.451722 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:54.451777 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:54.451903 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:54.453333 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:54.454020 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:54.456115 (+ 2095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:54.456417 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:54.456701 (+ 284us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:54.458489 (+ 1788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:54.458497 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:54.458504 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:50:54.458525 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:54.458557 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:50:54.458572 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:54.458670 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:54.458859 (+ 189us) server_negotiation.cc:299] Negotiation successful
0619 14:50:54.459048 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260619 14:50:54.460434 19242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:54.449026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45481 (local address 127.0.98.1:46005)
0619 14:50:54.449945 (+ 919us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:54.450004 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:54.451453 (+ 1449us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:54.452108 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:54.452116 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:54.452251 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:54.453134 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:54.453151 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:54.454196 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:54.454203 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:54.455776 (+ 1573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:54.455787 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:54.456811 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:54.456816 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:54.457063 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:54.457945 (+ 882us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:50:54.457947 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:54.457996 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:50:54.458260 (+ 264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:50:54.458682 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:50:54.458691 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:50:54.458696 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 14:50:54.458815 (+ 119us) client_negotiation.cc:240] Negotiation successful
0619 14:50:54.459147 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":134,"threads_started":1}
I20260619 14:50:54.462163 19240 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45481
I20260619 14:50:54.462560 19240 heartbeater.cc:464] Registering TS with master...
I20260619 14:50:54.463578 19240 heartbeater.cc:511] Master 127.0.98.62:45481 requested a full tablet report, sending...
I20260619 14:50:54.465530 19052 ts_manager.cc:194] Registered new tserver with Master: 1b1f27fbeaf54820b618993e0daac38c (127.0.98.1:37231)
I20260619 14:50:54.466876 19052 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:46005
W20260619 14:50:54.584987 19243 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:50:54.585358 19243 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:50:54.585407 19243 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:50:54.589805 19243 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:50:54.589890 19243 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:50:54.589952 19243 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 14:50:54.595335 19243 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880383132139-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880383132139-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880383132139-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45481
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19243
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880383132139-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:50:54.596587 19243 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:50:54.598120 19243 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:50:54.610133 19248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:54.610096 19249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:54.610113 19251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:50:54.611018 19243 server_base.cc:1034] running on GCE node
I20260619 14:50:54.611701 19243 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:50:54.612519 19243 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:50:54.613888 19243 hybrid_clock.cc:648] HybridClock initialized: now 1781880654613822 us; error 80 us; skew 500 ppm
I20260619 14:50:54.616811 19243 webserver.cc:460] Webserver started at http://127.0.98.2:45891/ using document root <none> and password file <none>
I20260619 14:50:54.617522 19243 fs_manager.cc:359] Metadata directory not provided
I20260619 14:50:54.617633 19243 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:50:54.617863 19243 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:50:54.620210 19243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880383132139-392-0/minicluster-data/ts-1/data/instance:
uuid: "0ee00117d72c48789826861740bda3d9"
format_stamp: "Formatted at 2026-06-19 14:50:54 on dist-test-slave-2sb1"
server_key: "5003ba9e416538a20b9e91f2152d8054"
server_key_iv: "a05d9da90c7da4e2ba2399642e17f70c"
server_key_version: "encryptionkey@0"
I20260619 14:50:54.620992 19243 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880383132139-392-0/minicluster-data/ts-1/wal/instance:
uuid: "0ee00117d72c48789826861740bda3d9"
format_stamp: "Formatted at 2026-06-19 14:50:54 on dist-test-slave-2sb1"
server_key: "5003ba9e416538a20b9e91f2152d8054"
server_key_iv: "a05d9da90c7da4e2ba2399642e17f70c"
server_key_version: "encryptionkey@0"
I20260619 14:50:54.625460 19243 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 14:50:54.628593 19257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:54.630118 19243 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:50:54.630234 19243 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880383132139-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880383132139-392-0/minicluster-data/ts-1/wal
uuid: "0ee00117d72c48789826861740bda3d9"
format_stamp: "Formatted at 2026-06-19 14:50:54 on dist-test-slave-2sb1"
server_key: "5003ba9e416538a20b9e91f2152d8054"
server_key_iv: "a05d9da90c7da4e2ba2399642e17f70c"
server_key_version: "encryptionkey@0"
I20260619 14:50:54.630342 19243 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880383132139-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880383132139-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880383132139-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:50:54.653826 19243 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:50:54.654608 19243 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:50:54.654778 19243 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:50:54.655349 19243 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:50:54.656752 19243 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:50:54.656812 19243 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:54.656864 19243 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:50:54.656883 19243 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:54.667797 19243 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40499
I20260619 14:50:54.667814 19370 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40499 every 8 connection(s)
I20260619 14:50:54.668862 19243 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880383132139-392-0/minicluster-data/ts-1/data/info.pb
I20260619 14:50:54.677372 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 19243
I20260619 14:50:54.677493 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880383132139-392-0/minicluster-data/ts-1/wal/instance
I20260619 14:50:54.677778 392 external_mini_cluster.cc:1428] Setting key 7a2990b46b4f128821b4bbd83f07aa7e
I20260619 14:50:54.680564 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880383132139-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880383132139-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880383132139-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880383132139-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45481
--builtin_ntp_servers=127.0.98.20:40849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 14:50:54.681063 19091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:54.671307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36625 (local address 127.0.98.62:45481)
0619 14:50:54.671584 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:54.671588 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:54.672669 (+ 1081us) server_negotiation.cc:407] Connection header received
0619 14:50:54.674274 (+ 1605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:54.674277 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:54.674330 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:54.674437 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:54.675608 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:54.676163 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:54.677994 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:54.678304 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:54.678432 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:54.680310 (+ 1878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:54.680314 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:54.680317 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:50:54.680337 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:54.680364 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:50:54.680376 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:54.680459 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:54.680771 (+ 312us) server_negotiation.cc:299] Negotiation successful
0619 14:50:54.680872 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190}
I20260619 14:50:54.683365 19373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:54.671743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45481 (local address 127.0.98.2:36625)
0619 14:50:54.672466 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:54.672517 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:54.674021 (+ 1504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:54.674749 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:54.674757 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:54.674857 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:54.675412 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:54.675428 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:54.676308 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:54.676311 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:54.677761 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:54.677775 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:54.678630 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:54.678643 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:54.678804 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:54.679716 (+ 912us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:50:54.679718 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:54.679779 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:50:54.679880 (+ 101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:50:54.680484 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:50:54.680495 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:50:54.680502 (+ 7us) client_negotiation.cc:769] Sending connection context
0619 14:50:54.680704 (+ 202us) client_negotiation.cc:240] Negotiation successful
0619 14:50:54.681955 (+ 1251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":147,"threads_started":1}
I20260619 14:50:54.685153 19371 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45481
I20260619 14:50:54.685906 19371 heartbeater.cc:464] Registering TS with master...
I20260619 14:50:54.686651 19371 heartbeater.cc:511] Master 127.0.98.62:45481 requested a full tablet report, sending...
I20260619 14:50:54.687930 19052 ts_manager.cc:194] Registered new tserver with Master: 0ee00117d72c48789826861740bda3d9 (127.0.98.2:40499)
I20260619 14:50:54.688774 19052 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:36625
W20260619 14:50:54.808387 19374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:50:54.808773 19374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:50:54.808859 19374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:50:54.813397 19374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:50:54.813491 19374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:50:54.813593 19374 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 14:50:54.818897 19374 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880383132139-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880383132139-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880383132139-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45481
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19374
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880383132139-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:50:54.820086 19374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:50:54.821609 19374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:50:54.833272 19382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:54.833272 19379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:54.833285 19380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:50:54.833824 19374 server_base.cc:1034] running on GCE node
I20260619 14:50:54.834334 19374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:50:54.835170 19374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:50:54.836496 19374 hybrid_clock.cc:648] HybridClock initialized: now 1781880654836466 us; error 45 us; skew 500 ppm
I20260619 14:50:54.838940 19374 webserver.cc:460] Webserver started at http://127.0.98.3:34229/ using document root <none> and password file <none>
I20260619 14:50:54.839660 19374 fs_manager.cc:359] Metadata directory not provided
I20260619 14:50:54.839740 19374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:50:54.840282 19374 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:50:54.843727 19374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880383132139-392-0/minicluster-data/ts-2/data/instance:
uuid: "cfa3ba80100a4debbd2dbdad050f5473"
format_stamp: "Formatted at 2026-06-19 14:50:54 on dist-test-slave-2sb1"
server_key: "0ae92da3dbb272f3c3a67ce6ab7de465"
server_key_iv: "2850d0db3b40cb1d574ced2d6ce32a37"
server_key_version: "encryptionkey@0"
I20260619 14:50:54.844437 19374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880383132139-392-0/minicluster-data/ts-2/wal/instance:
uuid: "cfa3ba80100a4debbd2dbdad050f5473"
format_stamp: "Formatted at 2026-06-19 14:50:54 on dist-test-slave-2sb1"
server_key: "0ae92da3dbb272f3c3a67ce6ab7de465"
server_key_iv: "2850d0db3b40cb1d574ced2d6ce32a37"
server_key_version: "encryptionkey@0"
I20260619 14:50:54.849184 19374 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 14:50:54.852378 19388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:54.854238 19374 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:50:54.854578 19374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880383132139-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880383132139-392-0/minicluster-data/ts-2/wal
uuid: "cfa3ba80100a4debbd2dbdad050f5473"
format_stamp: "Formatted at 2026-06-19 14:50:54 on dist-test-slave-2sb1"
server_key: "0ae92da3dbb272f3c3a67ce6ab7de465"
server_key_iv: "2850d0db3b40cb1d574ced2d6ce32a37"
server_key_version: "encryptionkey@0"
I20260619 14:50:54.854743 19374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880383132139-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880383132139-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880383132139-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:50:54.865824 19374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:50:54.866691 19374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:50:54.866932 19374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:50:54.867571 19374 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:50:54.868640 19374 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:50:54.868742 19374 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:54.868819 19374 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:50:54.868863 19374 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:54.878660 19374 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40887
I20260619 14:50:54.878682 19501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40887 every 8 connection(s)
I20260619 14:50:54.879655 19374 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880383132139-392-0/minicluster-data/ts-2/data/info.pb
I20260619 14:50:54.880024 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 19374
I20260619 14:50:54.880159 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880383132139-392-0/minicluster-data/ts-2/wal/instance
I20260619 14:50:54.880497 392 external_mini_cluster.cc:1428] Setting key 20c30789f19858d9e98c56cc8157ce4f
I20260619 14:50:54.891161 19091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:54.881940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53731 (local address 127.0.98.62:45481)
0619 14:50:54.882086 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:54.882090 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:54.883499 (+ 1409us) server_negotiation.cc:407] Connection header received
0619 14:50:54.885171 (+ 1672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:54.885174 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:54.885227 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:54.885319 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:54.886575 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:54.887199 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:54.888621 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:54.888920 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:54.889030 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:54.890580 (+ 1550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:54.890584 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:54.890587 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:50:54.890602 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:54.890621 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:50:54.890629 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:54.890691 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:54.890930 (+ 239us) server_negotiation.cc:299] Negotiation successful
0619 14:50:54.890994 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260619 14:50:54.892163 19504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:54.882499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45481 (local address 127.0.98.3:53731)
0619 14:50:54.883349 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:54.883392 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:54.884907 (+ 1515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:54.885575 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:54.885583 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:54.885679 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:54.886262 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:54.886285 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:54.887365 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:54.887368 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:54.888447 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:54.888459 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:54.889127 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:54.889131 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:54.889326 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:54.890173 (+ 847us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:50:54.890174 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:54.890220 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:50:54.890297 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:50:54.890742 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:50:54.890750 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:50:54.890755 (+ 5us) client_negotiation.cc:769] Sending connection context
0619 14:50:54.890909 (+ 154us) client_negotiation.cc:240] Negotiation successful
0619 14:50:54.891167 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":184,"threads_started":1}
I20260619 14:50:54.893996 19502 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45481
I20260619 14:50:54.894304 19502 heartbeater.cc:464] Registering TS with master...
I20260619 14:50:54.895159 19502 heartbeater.cc:511] Master 127.0.98.62:45481 requested a full tablet report, sending...
I20260619 14:50:54.896409 19052 ts_manager.cc:194] Registered new tserver with Master: cfa3ba80100a4debbd2dbdad050f5473 (127.0.98.3:40887)
I20260619 14:50:54.897131 19052 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:53731
I20260619 14:50:54.907886 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:50:54.914744 19091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:54.910384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57382 (local address 127.0.98.62:45481)
0619 14:50:54.910627 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:54.910631 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:54.910646 (+ 15us) server_negotiation.cc:407] Connection header received
0619 14:50:54.910691 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:54.910693 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:54.910734 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:54.910810 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:54.911195 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:54.911845 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:54.912816 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:54.913091 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:54.913177 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:54.914145 (+ 968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:54.914148 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:54.914150 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:50:54.914163 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:54.914182 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:50:54.914190 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:54.914298 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:54.914584 (+ 286us) server_negotiation.cc:299] Negotiation successful
0619 14:50:54.914624 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155}
I20260619 14:50:54.918383 19052 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57382:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:50:54.921517 19052 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:50:54.937336 19513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:54.930833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37231 (local address 127.0.0.1:44274)
0619 14:50:54.931336 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:54.931566 (+ 230us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:54.931702 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:54.932707 (+ 1005us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:54.932712 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:54.932732 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:50:54.932928 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:54.932938 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:54.934197 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:54.934204 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:54.935634 (+ 1430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:54.935644 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:54.936756 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:54.936760 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:54.936872 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:54.936923 (+ 51us) client_negotiation.cc:769] Sending connection context
0619 14:50:54.937047 (+ 124us) client_negotiation.cc:240] Negotiation successful
0619 14:50:54.937174 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"mutex_wait_us":3,"thread_start_us":158,"threads_started":1}
I20260619 14:50:54.937335 19514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:54.931016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44274 (local address 127.0.98.1:37231)
0619 14:50:54.932134 (+ 1118us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:54.932142 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:54.932169 (+ 27us) server_negotiation.cc:407] Connection header received
0619 14:50:54.932255 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:54.932262 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:54.932502 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:54.932716 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:50:54.933061 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:54.934088 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:54.935777 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:54.936623 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:54.936761 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:54.936959 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:54.937080 (+ 121us) server_negotiation.cc:299] Negotiation successful
0619 14:50:54.937167 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1005,"thread_start_us":153,"threads_started":1}
I20260619 14:50:54.937949 19515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:54.931719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40887 (local address 127.0.0.1:54302)
0619 14:50:54.932014 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:54.932028 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:54.932119 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:54.934054 (+ 1935us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:54.934057 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:54.934075 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:50:54.934153 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:54.934158 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:54.935410 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:54.935414 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:54.936364 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:54.936373 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:54.937390 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:54.937396 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:54.937645 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:54.937668 (+ 23us) client_negotiation.cc:769] Sending connection context
0619 14:50:54.937753 (+ 85us) client_negotiation.cc:240] Negotiation successful
0619 14:50:54.937822 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"mutex_wait_us":3,"thread_start_us":129,"threads_started":1}
I20260619 14:50:54.938074 19517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:54.931967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54302 (local address 127.0.98.3:40887)
0619 14:50:54.933468 (+ 1501us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:54.933479 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:54.933504 (+ 25us) server_negotiation.cc:407] Connection header received
0619 14:50:54.933574 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:54.933578 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:54.933746 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:54.933919 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:50:54.934298 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:54.935273 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:54.936487 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:54.937251 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:54.937370 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:54.937564 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:54.937795 (+ 231us) server_negotiation.cc:299] Negotiation successful
0619 14:50:54.937893 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1374,"thread_start_us":156,"threads_started":1}
I20260619 14:50:54.938381 19518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:54.931719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40499 (local address 127.0.0.1:57554)
0619 14:50:54.932399 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:54.932410 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:54.932474 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:54.933216 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:54.933219 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:54.933237 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:50:54.933324 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:54.933330 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:54.934659 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:54.934663 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:54.935773 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:54.935780 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:54.937931 (+ 2151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:54.937935 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:54.938027 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:54.938049 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 14:50:54.938128 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 14:50:54.938209 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"mutex_wait_us":472,"thread_start_us":87,"threads_started":1}
I20260619 14:50:54.938875 19516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:54.931961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57554 (local address 127.0.98.2:40499)
0619 14:50:54.932739 (+ 778us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:54.932750 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:54.932773 (+ 23us) server_negotiation.cc:407] Connection header received
0619 14:50:54.932845 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:54.932850 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:54.933067 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:54.933205 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:50:54.933453 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:54.934540 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:54.935892 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:54.936622 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:54.936761 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:54.936959 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:54.938600 (+ 1641us) server_negotiation.cc:299] Negotiation successful
0619 14:50:54.938707 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":658,"thread_start_us":92,"threads_started":1}
I20260619 14:50:54.940518 19174 tablet_service.cc:1467] Processing CreateTablet for tablet 0ec0262716ca47b7ba84853422a57afb (DEFAULT_TABLE table=test-table [id=52ed338abe4e456183505da9eb18baa0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:50:54.941591 19174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ec0262716ca47b7ba84853422a57afb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:50:54.941438 19436 tablet_service.cc:1467] Processing CreateTablet for tablet 0ec0262716ca47b7ba84853422a57afb (DEFAULT_TABLE table=test-table [id=52ed338abe4e456183505da9eb18baa0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:50:54.941438 19305 tablet_service.cc:1467] Processing CreateTablet for tablet 0ec0262716ca47b7ba84853422a57afb (DEFAULT_TABLE table=test-table [id=52ed338abe4e456183505da9eb18baa0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:50:54.942422 19305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ec0262716ca47b7ba84853422a57afb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:50:54.942417 19436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ec0262716ca47b7ba84853422a57afb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:50:54.948841 19519 tablet_bootstrap.cc:492] T 0ec0262716ca47b7ba84853422a57afb P 1b1f27fbeaf54820b618993e0daac38c: Bootstrap starting.
I20260619 14:50:54.949999 19520 tablet_bootstrap.cc:492] T 0ec0262716ca47b7ba84853422a57afb P cfa3ba80100a4debbd2dbdad050f5473: Bootstrap starting.
I20260619 14:50:54.950313 19521 tablet_bootstrap.cc:492] T 0ec0262716ca47b7ba84853422a57afb P 0ee00117d72c48789826861740bda3d9: Bootstrap starting.
I20260619 14:50:54.951325 19519 tablet_bootstrap.cc:654] T 0ec0262716ca47b7ba84853422a57afb P 1b1f27fbeaf54820b618993e0daac38c: Neither blocks nor log segments found. Creating new log.
I20260619 14:50:54.952088 19519 log.cc:824] T 0ec0262716ca47b7ba84853422a57afb P 1b1f27fbeaf54820b618993e0daac38c: Log is configured to *not* fsync() on all Append() calls
I20260619 14:50:54.952237 19521 tablet_bootstrap.cc:654] T 0ec0262716ca47b7ba84853422a57afb P 0ee00117d72c48789826861740bda3d9: Neither blocks nor log segments found. Creating new log.
I20260619 14:50:54.952237 19520 tablet_bootstrap.cc:654] T 0ec0262716ca47b7ba84853422a57afb P cfa3ba80100a4debbd2dbdad050f5473: Neither blocks nor log segments found. Creating new log.
I20260619 14:50:54.953130 19520 log.cc:824] T 0ec0262716ca47b7ba84853422a57afb P cfa3ba80100a4debbd2dbdad050f5473: Log is configured to *not* fsync() on all Append() calls
I20260619 14:50:54.953130 19521 log.cc:824] T 0ec0262716ca47b7ba84853422a57afb P 0ee00117d72c48789826861740bda3d9: Log is configured to *not* fsync() on all Append() calls
I20260619 14:50:54.954474 19519 tablet_bootstrap.cc:492] T 0ec0262716ca47b7ba84853422a57afb P 1b1f27fbeaf54820b618993e0daac38c: No bootstrap required, opened a new log
I20260619 14:50:54.954967 19519 ts_tablet_manager.cc:1397] T 0ec0262716ca47b7ba84853422a57afb P 1b1f27fbeaf54820b618993e0daac38c: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260619 14:50:54.955183 19520 tablet_bootstrap.cc:492] T 0ec0262716ca47b7ba84853422a57afb P cfa3ba80100a4debbd2dbdad050f5473: No bootstrap required, opened a new log
I20260619 14:50:54.955185 19521 tablet_bootstrap.cc:492] T 0ec0262716ca47b7ba84853422a57afb P 0ee00117d72c48789826861740bda3d9: No bootstrap required, opened a new log
I20260619 14:50:54.955376 19520 ts_tablet_manager.cc:1397] T 0ec0262716ca47b7ba84853422a57afb P cfa3ba80100a4debbd2dbdad050f5473: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260619 14:50:54.955376 19521 ts_tablet_manager.cc:1397] T 0ec0262716ca47b7ba84853422a57afb P 0ee00117d72c48789826861740bda3d9: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 14:50:54.958541 19519 raft_consensus.cc:348] T 0ec0262716ca47b7ba84853422a57afb P 1b1f27fbeaf54820b618993e0daac38c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b1f27fbeaf54820b618993e0daac38c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37231 } } peers { permanent_uuid: "cfa3ba80100a4debbd2dbdad050f5473" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40887 } } peers { permanent_uuid: "0ee00117d72c48789826861740bda3d9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40499 } }
I20260619 14:50:54.958811 19519 raft_consensus.cc:374] T 0ec0262716ca47b7ba84853422a57afb P 1b1f27fbeaf54820b618993e0daac38c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:50:54.958949 19519 raft_consensus.cc:729] T 0ec0262716ca47b7ba84853422a57afb P 1b1f27fbeaf54820b618993e0daac38c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b1f27fbeaf54820b618993e0daac38c, State: Initialized, Role: FOLLOWER
I20260619 14:50:54.959487 19519 consensus_queue.cc:260] T 0ec0262716ca47b7ba84853422a57afb P 1b1f27fbeaf54820b618993e0daac38c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b1f27fbeaf54820b618993e0daac38c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37231 } } peers { permanent_uuid: "cfa3ba80100a4debbd2dbdad050f5473" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40887 } } peers { permanent_uuid: "0ee00117d72c48789826861740bda3d9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40499 } }
I20260619 14:50:54.959828 19521 raft_consensus.cc:348] T 0ec0262716ca47b7ba84853422a57afb P 0ee00117d72c48789826861740bda3d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b1f27fbeaf54820b618993e0daac38c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37231 } } peers { permanent_uuid: "cfa3ba80100a4debbd2dbdad050f5473" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40887 } } peers { permanent_uuid: "0ee00117d72c48789826861740bda3d9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40499 } }
I20260619 14:50:54.959828 19520 raft_consensus.cc:348] T 0ec0262716ca47b7ba84853422a57afb P cfa3ba80100a4debbd2dbdad050f5473 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b1f27fbeaf54820b618993e0daac38c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37231 } } peers { permanent_uuid: "cfa3ba80100a4debbd2dbdad050f5473" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40887 } } peers { permanent_uuid: "0ee00117d72c48789826861740bda3d9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40499 } }
I20260619 14:50:54.960093 19521 raft_consensus.cc:374] T 0ec0262716ca47b7ba84853422a57afb P 0ee00117d72c48789826861740bda3d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:50:54.960125 19520 raft_consensus.cc:374] T 0ec0262716ca47b7ba84853422a57afb P cfa3ba80100a4debbd2dbdad050f5473 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:50:54.960172 19520 raft_consensus.cc:729] T 0ec0262716ca47b7ba84853422a57afb P cfa3ba80100a4debbd2dbdad050f5473 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfa3ba80100a4debbd2dbdad050f5473, State: Initialized, Role: FOLLOWER
I20260619 14:50:54.960167 19521 raft_consensus.cc:729] T 0ec0262716ca47b7ba84853422a57afb P 0ee00117d72c48789826861740bda3d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ee00117d72c48789826861740bda3d9, State: Initialized, Role: FOLLOWER
I20260619 14:50:54.960183 19240 heartbeater.cc:503] Master 127.0.98.62:45481 was elected leader, sending a full tablet report...
I20260619 14:50:54.960615 19521 consensus_queue.cc:260] T 0ec0262716ca47b7ba84853422a57afb P 0ee00117d72c48789826861740bda3d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b1f27fbeaf54820b618993e0daac38c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37231 } } peers { permanent_uuid: "cfa3ba80100a4debbd2dbdad050f5473" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40887 } } peers { permanent_uuid: "0ee00117d72c48789826861740bda3d9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40499 } }
I20260619 14:50:54.960611 19520 consensus_queue.cc:260] T 0ec0262716ca47b7ba84853422a57afb P cfa3ba80100a4debbd2dbdad050f5473 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b1f27fbeaf54820b618993e0daac38c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37231 } } peers { permanent_uuid: "cfa3ba80100a4debbd2dbdad050f5473" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40887 } } peers { permanent_uuid: "0ee00117d72c48789826861740bda3d9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40499 } }
I20260619 14:50:54.960717 19519 ts_tablet_manager.cc:1428] T 0ec0262716ca47b7ba84853422a57afb P 1b1f27fbeaf54820b618993e0daac38c: Time spent starting tablet: real 0.006s user 0.001s sys 0.005s
I20260619 14:50:54.961930 19520 ts_tablet_manager.cc:1428] T 0ec0262716ca47b7ba84853422a57afb P cfa3ba80100a4debbd2dbdad050f5473: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260619 14:50:54.961930 19521 ts_tablet_manager.cc:1428] T 0ec0262716ca47b7ba84853422a57afb P 0ee00117d72c48789826861740bda3d9: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260619 14:50:54.962321 19371 heartbeater.cc:503] Master 127.0.98.62:45481 was elected leader, sending a full tablet report...
I20260619 14:50:54.963615 19502 heartbeater.cc:503] Master 127.0.98.62:45481 was elected leader, sending a full tablet report...
I20260619 14:50:55.046314 19526 raft_consensus.cc:482] T 0ec0262716ca47b7ba84853422a57afb P 0ee00117d72c48789826861740bda3d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:50:55.046648 19526 raft_consensus.cc:504] T 0ec0262716ca47b7ba84853422a57afb P 0ee00117d72c48789826861740bda3d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b1f27fbeaf54820b618993e0daac38c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37231 } } peers { permanent_uuid: "cfa3ba80100a4debbd2dbdad050f5473" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40887 } } peers { permanent_uuid: "0ee00117d72c48789826861740bda3d9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40499 } }
I20260619 14:50:55.047999 19526 leader_election.cc:288] T 0ec0262716ca47b7ba84853422a57afb P 0ee00117d72c48789826861740bda3d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b1f27fbeaf54820b618993e0daac38c (127.0.98.1:37231), cfa3ba80100a4debbd2dbdad050f5473 (127.0.98.3:40887)
I20260619 14:50:55.052204 19373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:55.048219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37231 (local address 127.0.98.2:42747)
0619 14:50:55.048341 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:55.048358 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:55.048479 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:55.048851 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:55.048854 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:55.048871 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:50:55.048999 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:55.049006 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:55.049746 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:55.049752 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:55.050548 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:55.050557 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:55.051615 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:55.051623 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:55.051780 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:55.051817 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 14:50:55.051904 (+ 87us) client_negotiation.cc:240] Negotiation successful
0619 14:50:55.052013 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260619 14:50:55.052282 19514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:55.048345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42747 (local address 127.0.98.1:37231)
0619 14:50:55.048562 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:55.048569 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:55.048585 (+ 16us) server_negotiation.cc:407] Connection header received
0619 14:50:55.048664 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:55.048667 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:55.048741 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:55.048839 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:50:55.049125 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:55.049632 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:55.050685 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:55.051428 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:55.051536 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:55.051609 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:55.051950 (+ 341us) server_negotiation.cc:299] Negotiation successful
0619 14:50:55.052056 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
I20260619 14:50:55.052898 19517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:55.048547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:32859 (local address 127.0.98.3:40887)
0619 14:50:55.048687 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:55.048691 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:55.048917 (+ 226us) server_negotiation.cc:407] Connection header received
0619 14:50:55.049145 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:55.049148 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:55.049199 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:55.049288 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:50:55.049557 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:55.050071 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:55.051451 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:55.052124 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:55.052226 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:55.052286 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:55.052689 (+ 403us) server_negotiation.cc:299] Negotiation successful
0619 14:50:55.052778 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 14:50:55.053335 19528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:55.048335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40887 (local address 127.0.98.2:32859)
0619 14:50:55.048812 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:55.048824 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:55.048959 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:55.049300 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:55.049302 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:55.049318 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:50:55.049446 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:55.049452 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:55.050202 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:55.050208 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:55.051298 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:55.051311 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:55.052390 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:55.052394 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:55.052503 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:55.052531 (+ 28us) client_negotiation.cc:769] Sending connection context
0619 14:50:55.053077 (+ 546us) client_negotiation.cc:240] Negotiation successful
0619 14:50:55.053171 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":216,"threads_started":1}
I20260619 14:50:55.053341 19194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ec0262716ca47b7ba84853422a57afb" candidate_uuid: "0ee00117d72c48789826861740bda3d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b1f27fbeaf54820b618993e0daac38c" is_pre_election: true
I20260619 14:50:55.053711 19194 raft_consensus.cc:2454] T 0ec0262716ca47b7ba84853422a57afb P 1b1f27fbeaf54820b618993e0daac38c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ee00117d72c48789826861740bda3d9 in term 0.
I20260619 14:50:55.054592 19260 leader_election.cc:302] T 0ec0262716ca47b7ba84853422a57afb P 0ee00117d72c48789826861740bda3d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ee00117d72c48789826861740bda3d9, 1b1f27fbeaf54820b618993e0daac38c; no voters:
I20260619 14:50:55.054934 19526 raft_consensus.cc:2781] T 0ec0262716ca47b7ba84853422a57afb P 0ee00117d72c48789826861740bda3d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:50:55.054827 19456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ec0262716ca47b7ba84853422a57afb" candidate_uuid: "0ee00117d72c48789826861740bda3d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfa3ba80100a4debbd2dbdad050f5473" is_pre_election: true
I20260619 14:50:55.055146 19526 raft_consensus.cc:482] T 0ec0262716ca47b7ba84853422a57afb P 0ee00117d72c48789826861740bda3d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:50:55.055229 19526 raft_consensus.cc:3037] T 0ec0262716ca47b7ba84853422a57afb P 0ee00117d72c48789826861740bda3d9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:50:55.055269 19456 raft_consensus.cc:2454] T 0ec0262716ca47b7ba84853422a57afb P cfa3ba80100a4debbd2dbdad050f5473 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ee00117d72c48789826861740bda3d9 in term 0.
I20260619 14:50:55.056888 19526 raft_consensus.cc:504] T 0ec0262716ca47b7ba84853422a57afb P 0ee00117d72c48789826861740bda3d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b1f27fbeaf54820b618993e0daac38c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37231 } } peers { permanent_uuid: "cfa3ba80100a4debbd2dbdad050f5473" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40887 } } peers { permanent_uuid: "0ee00117d72c48789826861740bda3d9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40499 } }
I20260619 14:50:55.057319 19526 leader_election.cc:288] T 0ec0262716ca47b7ba84853422a57afb P 0ee00117d72c48789826861740bda3d9 [CANDIDATE]: Term 1 election: Requested vote from peers 1b1f27fbeaf54820b618993e0daac38c (127.0.98.1:37231), cfa3ba80100a4debbd2dbdad050f5473 (127.0.98.3:40887)
I20260619 14:50:55.057731 19194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ec0262716ca47b7ba84853422a57afb" candidate_uuid: "0ee00117d72c48789826861740bda3d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b1f27fbeaf54820b618993e0daac38c"
I20260619 14:50:55.057893 19194 raft_consensus.cc:3037] T 0ec0262716ca47b7ba84853422a57afb P 1b1f27fbeaf54820b618993e0daac38c [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:50:55.057873 19456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ec0262716ca47b7ba84853422a57afb" candidate_uuid: "0ee00117d72c48789826861740bda3d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfa3ba80100a4debbd2dbdad050f5473"
I20260619 14:50:55.058020 19456 raft_consensus.cc:3037] T 0ec0262716ca47b7ba84853422a57afb P cfa3ba80100a4debbd2dbdad050f5473 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:50:55.059374 19456 raft_consensus.cc:2454] T 0ec0262716ca47b7ba84853422a57afb P cfa3ba80100a4debbd2dbdad050f5473 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ee00117d72c48789826861740bda3d9 in term 1.
I20260619 14:50:55.059386 19194 raft_consensus.cc:2454] T 0ec0262716ca47b7ba84853422a57afb P 1b1f27fbeaf54820b618993e0daac38c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ee00117d72c48789826861740bda3d9 in term 1.
I20260619 14:50:55.059795 19260 leader_election.cc:302] T 0ec0262716ca47b7ba84853422a57afb P 0ee00117d72c48789826861740bda3d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0ee00117d72c48789826861740bda3d9, 1b1f27fbeaf54820b618993e0daac38c, cfa3ba80100a4debbd2dbdad050f5473; no voters:
I20260619 14:50:55.060065 19526 raft_consensus.cc:2781] T 0ec0262716ca47b7ba84853422a57afb P 0ee00117d72c48789826861740bda3d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:50:55.060398 19526 raft_consensus.cc:686] T 0ec0262716ca47b7ba84853422a57afb P 0ee00117d72c48789826861740bda3d9 [term 1 LEADER]: Becoming Leader. State: Replica: 0ee00117d72c48789826861740bda3d9, State: Running, Role: LEADER
I20260619 14:50:55.060846 19526 consensus_queue.cc:237] T 0ec0262716ca47b7ba84853422a57afb P 0ee00117d72c48789826861740bda3d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b1f27fbeaf54820b618993e0daac38c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37231 } } peers { permanent_uuid: "cfa3ba80100a4debbd2dbdad050f5473" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40887 } } peers { permanent_uuid: "0ee00117d72c48789826861740bda3d9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40499 } }
I20260619 14:50:55.064299 19050 catalog_manager.cc:5495] T 0ec0262716ca47b7ba84853422a57afb P 0ee00117d72c48789826861740bda3d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ee00117d72c48789826861740bda3d9 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "0ee00117d72c48789826861740bda3d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b1f27fbeaf54820b618993e0daac38c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cfa3ba80100a4debbd2dbdad050f5473" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ee00117d72c48789826861740bda3d9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40499 } health_report { overall_health: HEALTHY } } }
I20260619 14:50:55.097312 19516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:55.092994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57558 (local address 127.0.98.2:40499)
0619 14:50:55.093164 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:55.093170 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:55.093185 (+ 15us) server_negotiation.cc:407] Connection header received
0619 14:50:55.093273 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:55.093276 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:55.093333 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:55.093582 (+ 249us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:50:55.093868 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:55.094492 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:55.095522 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:55.095764 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:55.095889 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:55.096257 (+ 368us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:50:55.096797 (+ 540us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:50:55.096991 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:55.097098 (+ 107us) server_negotiation.cc:299] Negotiation successful
0619 14:50:55.097141 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260619 14:50:55.104141 19194 raft_consensus.cc:1261] T 0ec0262716ca47b7ba84853422a57afb P 1b1f27fbeaf54820b618993e0daac38c [term 1 FOLLOWER]: Refusing update from remote peer 0ee00117d72c48789826861740bda3d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:50:55.104406 19456 raft_consensus.cc:1261] T 0ec0262716ca47b7ba84853422a57afb P cfa3ba80100a4debbd2dbdad050f5473 [term 1 FOLLOWER]: Refusing update from remote peer 0ee00117d72c48789826861740bda3d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:50:55.105024 19529 consensus_queue.cc:1035] T 0ec0262716ca47b7ba84853422a57afb P 0ee00117d72c48789826861740bda3d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b1f27fbeaf54820b618993e0daac38c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:50:55.105242 19526 consensus_queue.cc:1035] T 0ec0262716ca47b7ba84853422a57afb P 0ee00117d72c48789826861740bda3d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfa3ba80100a4debbd2dbdad050f5473" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:50:55.113826 19532 mvcc.cc:205] Tried to move back new op lower bound from 7298583163298328576 to 7298583163131867136. Current Snapshot: MvccSnapshot[applied={T|T < 7298583163298328576}]
I20260619 14:50:55.115773 19534 mvcc.cc:205] Tried to move back new op lower bound from 7298583163298328576 to 7298583163131867136. Current Snapshot: MvccSnapshot[applied={T|T < 7298583163298328576}]
I20260619 14:50:55.124847 19050 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57382:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:50:55.125070 19050 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57382:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:50:55.128253 19050 catalog_manager.cc:5782] T 00000000000000000000000000000000 P df2790574e5c4d41a30d61fbea6777e9: Sending DeleteTablet for 3 replicas of tablet 0ec0262716ca47b7ba84853422a57afb
I20260619 14:50:55.129436 19436 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ec0262716ca47b7ba84853422a57afb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:50:55 UTC) from {username='slave'} at 127.0.0.1:54302
I20260619 14:50:55.129454 19174 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ec0262716ca47b7ba84853422a57afb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:50:55 UTC) from {username='slave'} at 127.0.0.1:44274
I20260619 14:50:55.129693 19305 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ec0262716ca47b7ba84853422a57afb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:50:55 UTC) from {username='slave'} at 127.0.0.1:57554
I20260619 14:50:55.130314 19542 tablet_replica.cc:330] T 0ec0262716ca47b7ba84853422a57afb P cfa3ba80100a4debbd2dbdad050f5473: stopping tablet replica
I20260619 14:50:55.130385 19543 tablet_replica.cc:330] T 0ec0262716ca47b7ba84853422a57afb P 0ee00117d72c48789826861740bda3d9: stopping tablet replica
I20260619 14:50:55.130714 19542 raft_consensus.cc:2229] T 0ec0262716ca47b7ba84853422a57afb P cfa3ba80100a4debbd2dbdad050f5473 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:50:55.130744 19543 raft_consensus.cc:2229] T 0ec0262716ca47b7ba84853422a57afb P 0ee00117d72c48789826861740bda3d9 [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:50:55.131135 19543 raft_consensus.cc:2258] T 0ec0262716ca47b7ba84853422a57afb P 0ee00117d72c48789826861740bda3d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:50:55.131204 19542 raft_consensus.cc:2258] T 0ec0262716ca47b7ba84853422a57afb P cfa3ba80100a4debbd2dbdad050f5473 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:50:55.131245 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 19112
I20260619 14:50:55.131466 19541 tablet_replica.cc:330] T 0ec0262716ca47b7ba84853422a57afb P 1b1f27fbeaf54820b618993e0daac38c: stopping tablet replica
I20260619 14:50:55.134128 19543 ts_tablet_manager.cc:1905] T 0ec0262716ca47b7ba84853422a57afb P 0ee00117d72c48789826861740bda3d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:50:55.134866 19542 ts_tablet_manager.cc:1905] T 0ec0262716ca47b7ba84853422a57afb P cfa3ba80100a4debbd2dbdad050f5473: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:50:55.137259 19543 ts_tablet_manager.cc:1918] T 0ec0262716ca47b7ba84853422a57afb P 0ee00117d72c48789826861740bda3d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:50:55.137391 19543 log.cc:1196] T 0ec0262716ca47b7ba84853422a57afb P 0ee00117d72c48789826861740bda3d9: Deleting WAL directory at /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880383132139-392-0/minicluster-data/ts-1/wal/wals/0ec0262716ca47b7ba84853422a57afb
I20260619 14:50:55.137768 19543 ts_tablet_manager.cc:1939] T 0ec0262716ca47b7ba84853422a57afb P 0ee00117d72c48789826861740bda3d9: Deleting consensus metadata
I20260619 14:50:55.138826 19542 ts_tablet_manager.cc:1918] T 0ec0262716ca47b7ba84853422a57afb P cfa3ba80100a4debbd2dbdad050f5473: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:50:55.139163 19542 log.cc:1196] T 0ec0262716ca47b7ba84853422a57afb P cfa3ba80100a4debbd2dbdad050f5473: Deleting WAL directory at /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781880383132139-392-0/minicluster-data/ts-2/wal/wals/0ec0262716ca47b7ba84853422a57afb
I20260619 14:50:55.139142 19038 catalog_manager.cc:4841] TS 0ee00117d72c48789826861740bda3d9 (127.0.98.2:40499): tablet 0ec0262716ca47b7ba84853422a57afb (table test-table [id=52ed338abe4e456183505da9eb18baa0]) successfully deleted
I20260619 14:50:55.139545 19542 ts_tablet_manager.cc:1939] T 0ec0262716ca47b7ba84853422a57afb P cfa3ba80100a4debbd2dbdad050f5473: Deleting consensus metadata
I20260619 14:50:55.141116 19037 catalog_manager.cc:4841] TS cfa3ba80100a4debbd2dbdad050f5473 (127.0.98.3:40887): tablet 0ec0262716ca47b7ba84853422a57afb (table test-table [id=52ed338abe4e456183505da9eb18baa0]) successfully deleted
W20260619 14:50:55.142215 19038 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37231 (error 108)
I20260619 14:50:55.143093 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 19243
I20260619 14:50:55.143386 19518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:55.143062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37231 (local address 127.0.0.1:44290)
0619 14:50:55.143203 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:55.143284 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260619 14:50:55.143743 19038 catalog_manager.cc:4584] TS 1b1f27fbeaf54820b618993e0daac38c (127.0.98.1:37231): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0ec0262716ca47b7ba84853422a57afb: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37231: connect: Connection refused (error 111)
I20260619 14:50:55.148450 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 19374
I20260619 14:50:55.154820 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 19021
2026-06-19T14:50:55Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1237 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880383132139-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:50:55 dist-test-slave-2sb1 krb5kdc[19547](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:50:55 dist-test-slave-2sb1 krb5kdc[19547](info): set up 2 sockets
Jun 19 14:50:55 dist-test-slave-2sb1 krb5kdc[19547](info): commencing operation
krb5kdc: starting...
W20260619 14:50:56.257248 392 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880383132139-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880383132139-392-0/krb5kdc/test-user.keytab.
Jun 19 14:50:56 dist-test-slave-2sb1 krb5kdc[19547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:50:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:50:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 14:50:56.397507 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880383132139-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880383132139-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880383132139-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38647
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38647
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:50:56.525964 19560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:50:56.526324 19560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:50:56.526409 19560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:50:56.531370 19560 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:50:56.531491 19560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:50:56.531533 19560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:50:56.531595 19560 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:50:56.531651 19560 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:50:56.537640 19560 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880383132139-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880383132139-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38647
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38647
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19560
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880383132139-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:50:56.538823 19560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:50:56.539887 19560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:50:56.549535 19565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:56.549539 19568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:50:56.549679 19560 server_base.cc:1034] running on GCE node
W20260619 14:50:56.549598 19566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:50:56.550565 19560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:50:56.551784 19560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:50:56.552982 19560 hybrid_clock.cc:648] HybridClock initialized: now 1781880656552949 us; error 49 us; skew 500 ppm
Jun 19 14:50:56 dist-test-slave-2sb1 krb5kdc[19547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880656, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:50:56.556308 19560 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 14:50:56.557541 19560 webserver.cc:460] Webserver started at http://127.0.98.62:45893/ using document root <none> and password file <none>
I20260619 14:50:56.558140 19560 fs_manager.cc:359] Metadata directory not provided
I20260619 14:50:56.558235 19560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:50:56.558496 19560 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:50:56.560704 19560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880383132139-392-0/minicluster-data/master-0/data/instance:
uuid: "a0cd44655d674b70ad95c298af92d2e9"
format_stamp: "Formatted at 2026-06-19 14:50:56 on dist-test-slave-2sb1"
server_key: "7a914eb5eefd6c026379d44c290733d2"
server_key_iv: "f7678131a624694016d79741ddf05235"
server_key_version: "encryptionkey@0"
I20260619 14:50:56.561321 19560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880383132139-392-0/minicluster-data/master-0/wal/instance:
uuid: "a0cd44655d674b70ad95c298af92d2e9"
format_stamp: "Formatted at 2026-06-19 14:50:56 on dist-test-slave-2sb1"
server_key: "7a914eb5eefd6c026379d44c290733d2"
server_key_iv: "f7678131a624694016d79741ddf05235"
server_key_version: "encryptionkey@0"
I20260619 14:50:56.565474 19560 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 14:50:56.568420 19575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:56.570030 19560 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260619 14:50:56.570199 19560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880383132139-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880383132139-392-0/minicluster-data/master-0/wal
uuid: "a0cd44655d674b70ad95c298af92d2e9"
format_stamp: "Formatted at 2026-06-19 14:50:56 on dist-test-slave-2sb1"
server_key: "7a914eb5eefd6c026379d44c290733d2"
server_key_iv: "f7678131a624694016d79741ddf05235"
server_key_version: "encryptionkey@0"
I20260619 14:50:56.570344 19560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880383132139-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880383132139-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880383132139-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:50:56.586633 19560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:50:56.591468 19560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:50:56.591694 19560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:50:56.601377 19560 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38647
I20260619 14:50:56.601431 19627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38647 every 8 connection(s)
I20260619 14:50:56.602860 19560 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:50:56.605376 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 19560
I20260619 14:50:56.605577 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880383132139-392-0/minicluster-data/master-0/wal/instance
I20260619 14:50:56.605929 392 external_mini_cluster.cc:1428] Setting key 50bb649fc4d746284953fe66032d19f8
I20260619 14:50:56.606850 19628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:50:56.616205 19628 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0cd44655d674b70ad95c298af92d2e9: Bootstrap starting.
Jun 19 14:50:56 dist-test-slave-2sb1 krb5kdc[19547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:50:56.619804 19628 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a0cd44655d674b70ad95c298af92d2e9: Neither blocks nor log segments found. Creating new log.
I20260619 14:50:56.620908 19628 log.cc:824] T 00000000000000000000000000000000 P a0cd44655d674b70ad95c298af92d2e9: Log is configured to *not* fsync() on all Append() calls
I20260619 14:50:56.623664 19628 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0cd44655d674b70ad95c298af92d2e9: No bootstrap required, opened a new log
I20260619 14:50:56.624774 19631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:56.607831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42124 (local address 127.0.98.62:38647)
0619 14:50:56.608404 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:56.608425 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:56.608487 (+ 62us) server_negotiation.cc:407] Connection header received
0619 14:50:56.609296 (+ 809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:56.609313 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:56.609703 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:56.610096 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:56.611047 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:56.612184 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:56.613562 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:56.614121 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:56.614329 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:56.617954 (+ 3625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:56.617963 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:56.617985 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:56.618031 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:56.620777 (+ 2746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:56.621355 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:56.621363 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:56.621370 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:56.621534 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:56.621971 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:56.621976 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:56.621980 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:56.622601 (+ 621us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:56.622904 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:56.623200 (+ 296us) server_negotiation.cc:299] Negotiation successful
0619 14:50:56.623556 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":178,"threads_started":1}
I20260619 14:50:56.629446 19628 raft_consensus.cc:348] T 00000000000000000000000000000000 P a0cd44655d674b70ad95c298af92d2e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0cd44655d674b70ad95c298af92d2e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38647 } }
I20260619 14:50:56.629820 19628 raft_consensus.cc:374] T 00000000000000000000000000000000 P a0cd44655d674b70ad95c298af92d2e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:50:56.629899 19628 raft_consensus.cc:729] T 00000000000000000000000000000000 P a0cd44655d674b70ad95c298af92d2e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0cd44655d674b70ad95c298af92d2e9, State: Initialized, Role: FOLLOWER
I20260619 14:50:56.630477 19628 consensus_queue.cc:260] T 00000000000000000000000000000000 P a0cd44655d674b70ad95c298af92d2e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0cd44655d674b70ad95c298af92d2e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38647 } }
I20260619 14:50:56.630618 19628 raft_consensus.cc:388] T 00000000000000000000000000000000 P a0cd44655d674b70ad95c298af92d2e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:50:56.630661 19628 raft_consensus.cc:482] T 00000000000000000000000000000000 P a0cd44655d674b70ad95c298af92d2e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:50:56.630738 19628 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a0cd44655d674b70ad95c298af92d2e9 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:50:56.631806 19628 raft_consensus.cc:504] T 00000000000000000000000000000000 P a0cd44655d674b70ad95c298af92d2e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0cd44655d674b70ad95c298af92d2e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38647 } }
I20260619 14:50:56.632136 19628 leader_election.cc:302] T 00000000000000000000000000000000 P a0cd44655d674b70ad95c298af92d2e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a0cd44655d674b70ad95c298af92d2e9; no voters:
I20260619 14:50:56.632421 19628 leader_election.cc:288] T 00000000000000000000000000000000 P a0cd44655d674b70ad95c298af92d2e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:50:56.632743 19633 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a0cd44655d674b70ad95c298af92d2e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:50:56.633358 19633 raft_consensus.cc:686] T 00000000000000000000000000000000 P a0cd44655d674b70ad95c298af92d2e9 [term 1 LEADER]: Becoming Leader. State: Replica: a0cd44655d674b70ad95c298af92d2e9, State: Running, Role: LEADER
I20260619 14:50:56.633788 19633 consensus_queue.cc:237] T 00000000000000000000000000000000 P a0cd44655d674b70ad95c298af92d2e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0cd44655d674b70ad95c298af92d2e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38647 } }
I20260619 14:50:56.634110 19628 sys_catalog.cc:564] T 00000000000000000000000000000000 P a0cd44655d674b70ad95c298af92d2e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:50:56.635608 19635 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0cd44655d674b70ad95c298af92d2e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a0cd44655d674b70ad95c298af92d2e9. Latest consensus state: current_term: 1 leader_uuid: "a0cd44655d674b70ad95c298af92d2e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0cd44655d674b70ad95c298af92d2e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38647 } } }
I20260619 14:50:56.635664 19634 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0cd44655d674b70ad95c298af92d2e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a0cd44655d674b70ad95c298af92d2e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0cd44655d674b70ad95c298af92d2e9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38647 } } }
I20260619 14:50:56.635794 19634 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0cd44655d674b70ad95c298af92d2e9 [sys.catalog]: This master's current role is: LEADER
I20260619 14:50:56.635746 19635 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0cd44655d674b70ad95c298af92d2e9 [sys.catalog]: This master's current role is: LEADER
I20260619 14:50:56.636250 19642 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:50:56.639591 19642 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:50:56.646755 19642 catalog_manager.cc:1347] Generated new cluster ID: 79a7bdf3afa740b995c3e37dd7f01f45
I20260619 14:50:56.646860 19642 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:50:56.657116 19642 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:50:56.658420 19642 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:50:56.669595 19642 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a0cd44655d674b70ad95c298af92d2e9: Generated new TSK 0
I20260619 14:50:56.670375 19642 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880383132139-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880383132139-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260619 14:50:56.718104 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880383132139-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880383132139-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880383132139-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880383132139-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38647
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880383132139-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:50:56.856050 19654 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:50:56.856340 19654 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:50:56.856385 19654 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:50:56.860482 19654 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:50:56.860556 19654 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:50:56.860616 19654 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 14:50:56.866361 19654 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880383132139-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880383132139-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880383132139-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880383132139-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38647
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19654
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880383132139-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:50:56.867695 19654 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:50:56.868866 19654 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:50:56.880198 19659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:50:56.880250 19654 server_base.cc:1034] running on GCE node
W20260619 14:50:56.880214 19662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:56.880191 19660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:50:56.881047 19654 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:50:56.881752 19654 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:50:56.882936 19654 hybrid_clock.cc:648] HybridClock initialized: now 1781880656882908 us; error 43 us; skew 500 ppm
Jun 19 14:50:56 dist-test-slave-2sb1 krb5kdc[19547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880656, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:50:56.886183 19654 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260619 14:50:56.887247 19654 webserver.cc:460] Webserver started at http://127.0.98.1:35917/ using document root <none> and password file <none>
I20260619 14:50:56.887876 19654 fs_manager.cc:359] Metadata directory not provided
I20260619 14:50:56.887946 19654 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:50:56.888176 19654 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:50:56.890640 19654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880383132139-392-0/minicluster-data/ts-0/data/instance:
uuid: "ae9bb40322b84d9990b6e4a969e49a59"
format_stamp: "Formatted at 2026-06-19 14:50:56 on dist-test-slave-2sb1"
server_key: "2d5a4a81bab3e86b63c4f51af775be28"
server_key_iv: "e7bce7921915643f1fbd5a1bb2e2e9a9"
server_key_version: "encryptionkey@0"
I20260619 14:50:56.891260 19654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880383132139-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ae9bb40322b84d9990b6e4a969e49a59"
format_stamp: "Formatted at 2026-06-19 14:50:56 on dist-test-slave-2sb1"
server_key: "2d5a4a81bab3e86b63c4f51af775be28"
server_key_iv: "e7bce7921915643f1fbd5a1bb2e2e9a9"
server_key_version: "encryptionkey@0"
I20260619 14:50:56.895538 19654 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 14:50:56.898258 19669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:56.899886 19654 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260619 14:50:56.900056 19654 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880383132139-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880383132139-392-0/minicluster-data/ts-0/wal
uuid: "ae9bb40322b84d9990b6e4a969e49a59"
format_stamp: "Formatted at 2026-06-19 14:50:56 on dist-test-slave-2sb1"
server_key: "2d5a4a81bab3e86b63c4f51af775be28"
server_key_iv: "e7bce7921915643f1fbd5a1bb2e2e9a9"
server_key_version: "encryptionkey@0"
I20260619 14:50:56.900202 19654 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880383132139-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880383132139-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880383132139-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:50:56.917395 19654 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:50:56.921388 19654 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:50:56.921608 19654 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:50:56.922219 19654 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:50:56.923063 19654 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:50:56.923110 19654 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:56.923178 19654 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:50:56.923208 19654 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:56.932368 19654 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36599
I20260619 14:50:56.932550 19782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36599 every 8 connection(s)
I20260619 14:50:56.933437 19654 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880383132139-392-0/minicluster-data/ts-0/data/info.pb
I20260619 14:50:56.936471 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 19654
I20260619 14:50:56.936638 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880383132139-392-0/minicluster-data/ts-0/wal/instance
I20260619 14:50:56.937079 392 external_mini_cluster.cc:1428] Setting key 077060ab9099c24149eedf30dd5f9402
Jun 19 14:50:56 dist-test-slave-2sb1 krb5kdc[19547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880656, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260619 14:50:56.950461 19631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:56.935714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60699 (local address 127.0.98.62:38647)
0619 14:50:56.935889 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:56.935893 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:56.937113 (+ 1220us) server_negotiation.cc:407] Connection header received
0619 14:50:56.938225 (+ 1112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:56.938228 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:56.938291 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:56.938375 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:56.939771 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:56.940371 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:56.942163 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:56.942526 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:56.942621 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:56.946127 (+ 3506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:56.946131 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:56.946134 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:56.946171 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:56.948013 (+ 1842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:56.948592 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:56.948595 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:56.948597 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:56.948744 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:56.949145 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:56.949148 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:56.949150 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:56.949477 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:56.949595 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:56.950126 (+ 531us) server_negotiation.cc:299] Negotiation successful
0619 14:50:56.950268 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260619 14:50:56.951335 19785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:56.936266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38647 (local address 127.0.98.1:60699)
0619 14:50:56.936938 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:56.936994 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:56.937996 (+ 1002us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:56.938583 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:56.938592 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:56.939115 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:56.939615 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:56.939628 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:56.940562 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:56.940568 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:56.941986 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:56.942004 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:56.942727 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:56.942741 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:56.943036 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:56.943838 (+ 802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:56.943868 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:56.945934 (+ 2066us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:50:56.948158 (+ 2224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:56.948164 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:56.948182 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:56.948463 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:56.948891 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:56.948893 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:56.948895 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:56.949038 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:56.949592 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:50:56.949600 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:50:56.949878 (+ 278us) client_negotiation.cc:769] Sending connection context
0619 14:50:56.950076 (+ 198us) client_negotiation.cc:240] Negotiation successful
0619 14:50:56.950425 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":180,"threads_started":1}
I20260619 14:50:56.952870 19783 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38647
I20260619 14:50:56.953171 19783 heartbeater.cc:464] Registering TS with master...
I20260619 14:50:56.953850 19783 heartbeater.cc:511] Master 127.0.98.62:38647 requested a full tablet report, sending...
I20260619 14:50:56.955811 19592 ts_manager.cc:194] Registered new tserver with Master: ae9bb40322b84d9990b6e4a969e49a59 (127.0.98.1:36599)
I20260619 14:50:56.957115 19592 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60699
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880383132139-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880383132139-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260619 14:50:56.973961 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880383132139-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880383132139-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880383132139-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880383132139-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38647
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880383132139-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:50:57.092522 19788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:50:57.092901 19788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:50:57.092986 19788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:50:57.097637 19788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:50:57.097771 19788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:50:57.097883 19788 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 14:50:57.103756 19788 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880383132139-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880383132139-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880383132139-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880383132139-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38647
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19788
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880383132139-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:50:57.105041 19788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:50:57.106117 19788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:50:57.116709 19796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:57.116636 19794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:57.116679 19793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:50:57.117345 19788 server_base.cc:1034] running on GCE node
I20260619 14:50:57.117885 19788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:50:57.118618 19788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:50:57.119819 19788 hybrid_clock.cc:648] HybridClock initialized: now 1781880657119773 us; error 64 us; skew 500 ppm
Jun 19 14:50:57 dist-test-slave-2sb1 krb5kdc[19547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880657, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:50:57.123476 19788 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260619 14:50:57.124521 19788 webserver.cc:460] Webserver started at http://127.0.98.2:46499/ using document root <none> and password file <none>
I20260619 14:50:57.125165 19788 fs_manager.cc:359] Metadata directory not provided
I20260619 14:50:57.125231 19788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:50:57.125411 19788 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:50:57.127483 19788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880383132139-392-0/minicluster-data/ts-1/data/instance:
uuid: "5862e1db953a49bca84a316949eaf9bf"
format_stamp: "Formatted at 2026-06-19 14:50:57 on dist-test-slave-2sb1"
server_key: "6329c7ee95aecc5bfa24a9c8434ed755"
server_key_iv: "6fe354e13b87a763667e7e65c8834241"
server_key_version: "encryptionkey@0"
I20260619 14:50:57.128000 19788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880383132139-392-0/minicluster-data/ts-1/wal/instance:
uuid: "5862e1db953a49bca84a316949eaf9bf"
format_stamp: "Formatted at 2026-06-19 14:50:57 on dist-test-slave-2sb1"
server_key: "6329c7ee95aecc5bfa24a9c8434ed755"
server_key_iv: "6fe354e13b87a763667e7e65c8834241"
server_key_version: "encryptionkey@0"
I20260619 14:50:57.132113 19788 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 14:50:57.135043 19803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:57.136873 19788 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:50:57.137091 19788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880383132139-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880383132139-392-0/minicluster-data/ts-1/wal
uuid: "5862e1db953a49bca84a316949eaf9bf"
format_stamp: "Formatted at 2026-06-19 14:50:57 on dist-test-slave-2sb1"
server_key: "6329c7ee95aecc5bfa24a9c8434ed755"
server_key_iv: "6fe354e13b87a763667e7e65c8834241"
server_key_version: "encryptionkey@0"
I20260619 14:50:57.137223 19788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880383132139-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880383132139-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880383132139-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:50:57.164399 19788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:50:57.168547 19788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:50:57.168872 19788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:50:57.169569 19788 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:50:57.170568 19788 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:50:57.170620 19788 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:57.170662 19788 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:50:57.170678 19788 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:57.180511 19788 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38025
I20260619 14:50:57.180529 19916 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38025 every 8 connection(s)
I20260619 14:50:57.181499 19788 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880383132139-392-0/minicluster-data/ts-1/data/info.pb
I20260619 14:50:57.181908 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 19788
I20260619 14:50:57.182055 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880383132139-392-0/minicluster-data/ts-1/wal/instance
I20260619 14:50:57.182379 392 external_mini_cluster.cc:1428] Setting key 4903edc4bf84e671d00e83e26964fd7f
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Jun 19 14:50:57 dist-test-slave-2sb1 krb5kdc[19547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880657, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:50:57.201584 19631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:57.183831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53077 (local address 127.0.98.62:38647)
0619 14:50:57.184020 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:57.184026 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:57.185374 (+ 1348us) server_negotiation.cc:407] Connection header received
0619 14:50:57.186524 (+ 1150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:57.186527 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:57.186589 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:57.186687 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:57.188163 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:57.189228 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:57.190506 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:57.190935 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:57.191090 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:57.197194 (+ 6104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:57.197202 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:57.197209 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:57.197253 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:57.199190 (+ 1937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:57.199854 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:57.199858 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:57.199860 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:57.199981 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:57.200366 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:57.200369 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:57.200371 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:57.200571 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:57.200702 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:57.201268 (+ 566us) server_negotiation.cc:299] Negotiation successful
0619 14:50:57.201416 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260619 14:50:57.202453 19919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:57.184347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38647 (local address 127.0.98.2:53077)
0619 14:50:57.185165 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:57.185236 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:57.186291 (+ 1055us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:57.186877 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:57.186884 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:57.187269 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:57.187961 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:57.187984 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:57.189439 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:57.189442 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:57.190363 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:57.190372 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:57.191114 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:57.191117 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:57.191284 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:57.192089 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:57.192105 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:57.196970 (+ 4865us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:50:57.199340 (+ 2370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:57.199348 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:57.199368 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:57.199723 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:57.200095 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:57.200099 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:57.200101 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:57.200256 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:57.200717 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:50:57.200725 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:50:57.201012 (+ 287us) client_negotiation.cc:769] Sending connection context
0619 14:50:57.201223 (+ 211us) client_negotiation.cc:240] Negotiation successful
0619 14:50:57.201578 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":210,"threads_started":1}
I20260619 14:50:57.204072 19917 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38647
I20260619 14:50:57.204380 19917 heartbeater.cc:464] Registering TS with master...
I20260619 14:50:57.205044 19917 heartbeater.cc:511] Master 127.0.98.62:38647 requested a full tablet report, sending...
I20260619 14:50:57.206276 19592 ts_manager.cc:194] Registered new tserver with Master: 5862e1db953a49bca84a316949eaf9bf (127.0.98.2:38025)
I20260619 14:50:57.206907 19592 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53077
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880383132139-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880383132139-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260619 14:50:57.217768 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880383132139-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880383132139-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880383132139-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880383132139-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38647
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880383132139-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:50:57.336593 19922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:50:57.336905 19922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:50:57.336980 19922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:50:57.341408 19922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:50:57.341480 19922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:50:57.341574 19922 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 14:50:57.347848 19922 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880383132139-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880383132139-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880383132139-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880383132139-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38647
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19922
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880383132139-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:50:57.348997 19922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:50:57.350021 19922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:50:57.360692 19927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:50:57.360944 19922 server_base.cc:1034] running on GCE node
W20260619 14:50:57.360673 19930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:57.360719 19928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:50:57.361729 19922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:50:57.362509 19922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:50:57.363703 19922 hybrid_clock.cc:648] HybridClock initialized: now 1781880657363647 us; error 71 us; skew 500 ppm
Jun 19 14:50:57 dist-test-slave-2sb1 krb5kdc[19547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880657, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:50:57.367352 19922 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260619 14:50:57.368502 19922 webserver.cc:460] Webserver started at http://127.0.98.3:34933/ using document root <none> and password file <none>
I20260619 14:50:57.369238 19922 fs_manager.cc:359] Metadata directory not provided
I20260619 14:50:57.369345 19922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:50:57.369591 19922 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:50:57.371886 19922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880383132139-392-0/minicluster-data/ts-2/data/instance:
uuid: "bcea9027465843a18380a2bebe5ac03f"
format_stamp: "Formatted at 2026-06-19 14:50:57 on dist-test-slave-2sb1"
server_key: "a8a98b49e6eaabb9fa15ed7e08407886"
server_key_iv: "d4a55e360d3d3cb11816022052edc746"
server_key_version: "encryptionkey@0"
I20260619 14:50:57.372510 19922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880383132139-392-0/minicluster-data/ts-2/wal/instance:
uuid: "bcea9027465843a18380a2bebe5ac03f"
format_stamp: "Formatted at 2026-06-19 14:50:57 on dist-test-slave-2sb1"
server_key: "a8a98b49e6eaabb9fa15ed7e08407886"
server_key_iv: "d4a55e360d3d3cb11816022052edc746"
server_key_version: "encryptionkey@0"
I20260619 14:50:57.376947 19922 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260619 14:50:57.380043 19937 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:57.381634 19922 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:50:57.381819 19922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880383132139-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880383132139-392-0/minicluster-data/ts-2/wal
uuid: "bcea9027465843a18380a2bebe5ac03f"
format_stamp: "Formatted at 2026-06-19 14:50:57 on dist-test-slave-2sb1"
server_key: "a8a98b49e6eaabb9fa15ed7e08407886"
server_key_iv: "d4a55e360d3d3cb11816022052edc746"
server_key_version: "encryptionkey@0"
I20260619 14:50:57.381969 19922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880383132139-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880383132139-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880383132139-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:50:57.423130 19922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:50:57.426915 19922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:50:57.427151 19922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:50:57.427772 19922 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:50:57.428620 19922 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:50:57.428745 19922 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:57.428824 19922 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:50:57.428882 19922 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:57.438720 19922 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39681
I20260619 14:50:57.438792 20050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39681 every 8 connection(s)
I20260619 14:50:57.439697 19922 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880383132139-392-0/minicluster-data/ts-2/data/info.pb
I20260619 14:50:57.445991 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 19922
I20260619 14:50:57.446180 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880383132139-392-0/minicluster-data/ts-2/wal/instance
I20260619 14:50:57.446491 392 external_mini_cluster.cc:1428] Setting key 8283a163ccc08193d03fc754226a52ac
Jun 19 14:50:57 dist-test-slave-2sb1 krb5kdc[19547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880657, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:50:57.455102 19631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:57.441822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33335 (local address 127.0.98.62:38647)
0619 14:50:57.441960 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:57.441963 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:57.443012 (+ 1049us) server_negotiation.cc:407] Connection header received
0619 14:50:57.443948 (+ 936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:57.443951 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:57.444009 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:57.444092 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:57.445330 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:57.445864 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:57.447352 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:57.447621 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:57.447734 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:57.451051 (+ 3317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:57.451060 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:57.451063 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:57.451090 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:57.452818 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:57.453407 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:57.453415 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:57.453416 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:57.453494 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:57.453881 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:57.453886 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:57.453888 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:57.454130 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:57.454239 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:57.454787 (+ 548us) server_negotiation.cc:299] Negotiation successful
0619 14:50:57.454929 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 14:50:57.456017 20053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:57.442006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38647 (local address 127.0.98.3:33335)
0619 14:50:57.442829 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:57.442893 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:57.443735 (+ 842us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:57.444271 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:57.444277 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:57.444666 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:57.445172 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:57.445185 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:57.446015 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:57.446019 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:57.447223 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:57.447233 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:57.447811 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:57.447816 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:57.448040 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:57.448869 (+ 829us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:50:57.448924 (+ 55us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:57.450827 (+ 1903us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:50:57.452952 (+ 2125us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:57.452959 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:57.452977 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:57.453280 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:57.453590 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:50:57.453593 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:50:57.453595 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:50:57.453749 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:50:57.454239 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:50:57.454258 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:50:57.454535 (+ 277us) client_negotiation.cc:769] Sending connection context
0619 14:50:57.454729 (+ 194us) client_negotiation.cc:240] Negotiation successful
0619 14:50:57.455085 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":113,"threads_started":1}
I20260619 14:50:57.457384 20051 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38647
I20260619 14:50:57.457669 20051 heartbeater.cc:464] Registering TS with master...
I20260619 14:50:57.458220 20051 heartbeater.cc:511] Master 127.0.98.62:38647 requested a full tablet report, sending...
I20260619 14:50:57.459522 19592 ts_manager.cc:194] Registered new tserver with Master: bcea9027465843a18380a2bebe5ac03f (127.0.98.3:39681)
I20260619 14:50:57.460188 19592 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33335
I20260619 14:50:57.460968 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:50:57.471474 19631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:57.463008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42132 (local address 127.0.98.62:38647)
0619 14:50:57.463182 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:57.463187 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:57.463330 (+ 143us) server_negotiation.cc:407] Connection header received
0619 14:50:57.463495 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:57.463498 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:57.463557 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:57.463648 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:57.464580 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:57.465222 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:57.466203 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:57.466418 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:57.466535 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:57.467999 (+ 1464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:57.468002 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:57.468004 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:50:57.468028 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:57.469635 (+ 1607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:57.470089 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:57.470093 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:57.470094 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:57.470179 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:50:57.470495 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:50:57.470498 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:50:57.470500 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:50:57.470733 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:57.470937 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:57.471182 (+ 245us) server_negotiation.cc:299] Negotiation successful
0619 14:50:57.471302 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260619 14:50:57.475695 19591 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42132:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:50:57.479205 19591 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:50:57.496727 20062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:57.490001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39681 (local address 127.0.0.1:54060)
0619 14:50:57.490781 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:57.490796 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:57.490941 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:57.491604 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:57.491610 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:57.491628 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:50:57.491777 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:57.491782 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:57.493218 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:57.493222 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:57.494843 (+ 1621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:57.494864 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:57.496113 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:57.496117 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:57.496222 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:57.496279 (+ 57us) client_negotiation.cc:769] Sending connection context
0619 14:50:57.496380 (+ 101us) client_negotiation.cc:240] Negotiation successful
0619 14:50:57.496518 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":722,"mutex_wait_us":2,"thread_start_us":153,"threads_started":1}
I20260619 14:50:57.497239 20064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:57.489833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36599 (local address 127.0.0.1:36326)
0619 14:50:57.490441 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:57.490499 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:57.490636 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:57.491770 (+ 1134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:57.491780 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:57.491797 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:50:57.491998 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:57.492007 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:57.493827 (+ 1820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:57.493831 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:57.495062 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:57.495070 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:57.495981 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:57.495985 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:57.496332 (+ 347us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:57.496380 (+ 48us) client_negotiation.cc:769] Sending connection context
0619 14:50:57.496459 (+ 79us) client_negotiation.cc:240] Negotiation successful
0619 14:50:57.497090 (+ 631us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"mutex_wait_us":4,"thread_start_us":99,"threads_started":1}
I20260619 14:50:57.497252 20063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:57.489949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36326 (local address 127.0.98.1:36599)
0619 14:50:57.490620 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:57.490628 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:57.490655 (+ 27us) server_negotiation.cc:407] Connection header received
0619 14:50:57.490887 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:57.490894 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:57.491125 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:57.491286 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:50:57.492205 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:57.493693 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:57.495200 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:57.495822 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:57.495969 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:57.496182 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:57.496771 (+ 589us) server_negotiation.cc:299] Negotiation successful
0619 14:50:57.497100 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":92,"threads_started":1}
I20260619 14:50:57.497861 20066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:57.491264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38025 (local address 127.0.0.1:41012)
0619 14:50:57.491521 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:57.491537 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:57.491616 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:57.493194 (+ 1578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:57.493197 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:57.493217 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:50:57.493371 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:57.493377 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:57.494598 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:57.494605 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:57.495865 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:57.495877 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:57.497394 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:57.497400 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:57.497512 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:57.497534 (+ 22us) client_negotiation.cc:769] Sending connection context
0619 14:50:57.497638 (+ 104us) client_negotiation.cc:240] Negotiation successful
0619 14:50:57.497721 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"mutex_wait_us":37,"thread_start_us":135,"threads_started":1}
I20260619 14:50:57.497958 20067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:57.491448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41012 (local address 127.0.98.2:38025)
0619 14:50:57.492359 (+ 911us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:57.492374 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:57.492400 (+ 26us) server_negotiation.cc:407] Connection header received
0619 14:50:57.492491 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:57.492496 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:57.492774 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:57.492997 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:50:57.493505 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:57.494453 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:57.496038 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:57.497028 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:57.497140 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:57.497301 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:57.497690 (+ 389us) server_negotiation.cc:299] Negotiation successful
0619 14:50:57.497797 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":753,"thread_start_us":126,"threads_started":1}
I20260619 14:50:57.498117 20065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:57.490652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54060 (local address 127.0.98.3:39681)
0619 14:50:57.491124 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:57.491136 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:57.491158 (+ 22us) server_negotiation.cc:407] Connection header received
0619 14:50:57.491236 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:57.491241 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:57.491425 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:57.491583 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:50:57.491924 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:57.493017 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:57.494997 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:57.495939 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:57.497275 (+ 1336us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:57.497585 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:57.497643 (+ 58us) server_negotiation.cc:299] Negotiation successful
0619 14:50:57.497749 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":186,"threads_started":1}
I20260619 14:50:57.500850 19717 tablet_service.cc:1467] Processing CreateTablet for tablet da949129e2cc4941bc51b6ad2971f213 (DEFAULT_TABLE table=test-table [id=1ce95aab4abe4d2fbcc99cd701ca9429]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:50:57.500888 19985 tablet_service.cc:1467] Processing CreateTablet for tablet da949129e2cc4941bc51b6ad2971f213 (DEFAULT_TABLE table=test-table [id=1ce95aab4abe4d2fbcc99cd701ca9429]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:50:57.500859 19851 tablet_service.cc:1467] Processing CreateTablet for tablet da949129e2cc4941bc51b6ad2971f213 (DEFAULT_TABLE table=test-table [id=1ce95aab4abe4d2fbcc99cd701ca9429]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:50:57.502063 19985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da949129e2cc4941bc51b6ad2971f213. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:50:57.502120 19717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da949129e2cc4941bc51b6ad2971f213. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:50:57.502120 19851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da949129e2cc4941bc51b6ad2971f213. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:50:57.509778 20068 tablet_bootstrap.cc:492] T da949129e2cc4941bc51b6ad2971f213 P bcea9027465843a18380a2bebe5ac03f: Bootstrap starting.
I20260619 14:50:57.511044 20070 tablet_bootstrap.cc:492] T da949129e2cc4941bc51b6ad2971f213 P 5862e1db953a49bca84a316949eaf9bf: Bootstrap starting.
I20260619 14:50:57.512866 20068 tablet_bootstrap.cc:654] T da949129e2cc4941bc51b6ad2971f213 P bcea9027465843a18380a2bebe5ac03f: Neither blocks nor log segments found. Creating new log.
I20260619 14:50:57.513054 20069 tablet_bootstrap.cc:492] T da949129e2cc4941bc51b6ad2971f213 P ae9bb40322b84d9990b6e4a969e49a59: Bootstrap starting.
I20260619 14:50:57.513411 20070 tablet_bootstrap.cc:654] T da949129e2cc4941bc51b6ad2971f213 P 5862e1db953a49bca84a316949eaf9bf: Neither blocks nor log segments found. Creating new log.
I20260619 14:50:57.514070 20068 log.cc:824] T da949129e2cc4941bc51b6ad2971f213 P bcea9027465843a18380a2bebe5ac03f: Log is configured to *not* fsync() on all Append() calls
I20260619 14:50:57.514719 20070 log.cc:824] T da949129e2cc4941bc51b6ad2971f213 P 5862e1db953a49bca84a316949eaf9bf: Log is configured to *not* fsync() on all Append() calls
I20260619 14:50:57.515571 20069 tablet_bootstrap.cc:654] T da949129e2cc4941bc51b6ad2971f213 P ae9bb40322b84d9990b6e4a969e49a59: Neither blocks nor log segments found. Creating new log.
I20260619 14:50:57.516222 20068 tablet_bootstrap.cc:492] T da949129e2cc4941bc51b6ad2971f213 P bcea9027465843a18380a2bebe5ac03f: No bootstrap required, opened a new log
I20260619 14:50:57.516438 20068 ts_tablet_manager.cc:1397] T da949129e2cc4941bc51b6ad2971f213 P bcea9027465843a18380a2bebe5ac03f: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.003s
I20260619 14:50:57.516520 20069 log.cc:824] T da949129e2cc4941bc51b6ad2971f213 P ae9bb40322b84d9990b6e4a969e49a59: Log is configured to *not* fsync() on all Append() calls
I20260619 14:50:57.517050 20070 tablet_bootstrap.cc:492] T da949129e2cc4941bc51b6ad2971f213 P 5862e1db953a49bca84a316949eaf9bf: No bootstrap required, opened a new log
I20260619 14:50:57.517287 20070 ts_tablet_manager.cc:1397] T da949129e2cc4941bc51b6ad2971f213 P 5862e1db953a49bca84a316949eaf9bf: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260619 14:50:57.519114 20069 tablet_bootstrap.cc:492] T da949129e2cc4941bc51b6ad2971f213 P ae9bb40322b84d9990b6e4a969e49a59: No bootstrap required, opened a new log
I20260619 14:50:57.519425 20069 ts_tablet_manager.cc:1397] T da949129e2cc4941bc51b6ad2971f213 P ae9bb40322b84d9990b6e4a969e49a59: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260619 14:50:57.520213 20068 raft_consensus.cc:348] T da949129e2cc4941bc51b6ad2971f213 P bcea9027465843a18380a2bebe5ac03f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae9bb40322b84d9990b6e4a969e49a59" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36599 } } peers { permanent_uuid: "bcea9027465843a18380a2bebe5ac03f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39681 } } peers { permanent_uuid: "5862e1db953a49bca84a316949eaf9bf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38025 } }
I20260619 14:50:57.520489 20068 raft_consensus.cc:374] T da949129e2cc4941bc51b6ad2971f213 P bcea9027465843a18380a2bebe5ac03f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:50:57.520550 20068 raft_consensus.cc:729] T da949129e2cc4941bc51b6ad2971f213 P bcea9027465843a18380a2bebe5ac03f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcea9027465843a18380a2bebe5ac03f, State: Initialized, Role: FOLLOWER
I20260619 14:50:57.521070 20068 consensus_queue.cc:260] T da949129e2cc4941bc51b6ad2971f213 P bcea9027465843a18380a2bebe5ac03f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae9bb40322b84d9990b6e4a969e49a59" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36599 } } peers { permanent_uuid: "bcea9027465843a18380a2bebe5ac03f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39681 } } peers { permanent_uuid: "5862e1db953a49bca84a316949eaf9bf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38025 } }
I20260619 14:50:57.521812 20051 heartbeater.cc:503] Master 127.0.98.62:38647 was elected leader, sending a full tablet report...
I20260619 14:50:57.522133 20068 ts_tablet_manager.cc:1428] T da949129e2cc4941bc51b6ad2971f213 P bcea9027465843a18380a2bebe5ac03f: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
I20260619 14:50:57.521920 20070 raft_consensus.cc:348] T da949129e2cc4941bc51b6ad2971f213 P 5862e1db953a49bca84a316949eaf9bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae9bb40322b84d9990b6e4a969e49a59" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36599 } } peers { permanent_uuid: "bcea9027465843a18380a2bebe5ac03f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39681 } } peers { permanent_uuid: "5862e1db953a49bca84a316949eaf9bf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38025 } }
I20260619 14:50:57.522224 20070 raft_consensus.cc:374] T da949129e2cc4941bc51b6ad2971f213 P 5862e1db953a49bca84a316949eaf9bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:50:57.522298 20070 raft_consensus.cc:729] T da949129e2cc4941bc51b6ad2971f213 P 5862e1db953a49bca84a316949eaf9bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5862e1db953a49bca84a316949eaf9bf, State: Initialized, Role: FOLLOWER
I20260619 14:50:57.522804 20070 consensus_queue.cc:260] T da949129e2cc4941bc51b6ad2971f213 P 5862e1db953a49bca84a316949eaf9bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae9bb40322b84d9990b6e4a969e49a59" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36599 } } peers { permanent_uuid: "bcea9027465843a18380a2bebe5ac03f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39681 } } peers { permanent_uuid: "5862e1db953a49bca84a316949eaf9bf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38025 } }
I20260619 14:50:57.523628 19917 heartbeater.cc:503] Master 127.0.98.62:38647 was elected leader, sending a full tablet report...
I20260619 14:50:57.524032 20070 ts_tablet_manager.cc:1428] T da949129e2cc4941bc51b6ad2971f213 P 5862e1db953a49bca84a316949eaf9bf: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260619 14:50:57.525257 20069 raft_consensus.cc:348] T da949129e2cc4941bc51b6ad2971f213 P ae9bb40322b84d9990b6e4a969e49a59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae9bb40322b84d9990b6e4a969e49a59" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36599 } } peers { permanent_uuid: "bcea9027465843a18380a2bebe5ac03f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39681 } } peers { permanent_uuid: "5862e1db953a49bca84a316949eaf9bf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38025 } }
I20260619 14:50:57.525508 20069 raft_consensus.cc:374] T da949129e2cc4941bc51b6ad2971f213 P ae9bb40322b84d9990b6e4a969e49a59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:50:57.525564 20069 raft_consensus.cc:729] T da949129e2cc4941bc51b6ad2971f213 P ae9bb40322b84d9990b6e4a969e49a59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae9bb40322b84d9990b6e4a969e49a59, State: Initialized, Role: FOLLOWER
I20260619 14:50:57.526028 20069 consensus_queue.cc:260] T da949129e2cc4941bc51b6ad2971f213 P ae9bb40322b84d9990b6e4a969e49a59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae9bb40322b84d9990b6e4a969e49a59" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36599 } } peers { permanent_uuid: "bcea9027465843a18380a2bebe5ac03f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39681 } } peers { permanent_uuid: "5862e1db953a49bca84a316949eaf9bf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38025 } }
I20260619 14:50:57.526773 19783 heartbeater.cc:503] Master 127.0.98.62:38647 was elected leader, sending a full tablet report...
I20260619 14:50:57.527092 20069 ts_tablet_manager.cc:1428] T da949129e2cc4941bc51b6ad2971f213 P ae9bb40322b84d9990b6e4a969e49a59: Time spent starting tablet: real 0.008s user 0.002s sys 0.004s
I20260619 14:50:57.629855 20076 raft_consensus.cc:482] T da949129e2cc4941bc51b6ad2971f213 P ae9bb40322b84d9990b6e4a969e49a59 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:50:57.630136 20076 raft_consensus.cc:504] T da949129e2cc4941bc51b6ad2971f213 P ae9bb40322b84d9990b6e4a969e49a59 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae9bb40322b84d9990b6e4a969e49a59" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36599 } } peers { permanent_uuid: "bcea9027465843a18380a2bebe5ac03f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39681 } } peers { permanent_uuid: "5862e1db953a49bca84a316949eaf9bf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38025 } }
I20260619 14:50:57.631501 20076 leader_election.cc:288] T da949129e2cc4941bc51b6ad2971f213 P ae9bb40322b84d9990b6e4a969e49a59 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bcea9027465843a18380a2bebe5ac03f (127.0.98.3:39681), 5862e1db953a49bca84a316949eaf9bf (127.0.98.2:38025)
I20260619 14:50:57.635756 20065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:57.631929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48113 (local address 127.0.98.3:39681)
0619 14:50:57.632119 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:57.632126 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:57.632242 (+ 116us) server_negotiation.cc:407] Connection header received
0619 14:50:57.632410 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:57.632413 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:57.632467 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:57.632608 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:50:57.632918 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:57.633421 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:57.634674 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:57.635171 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:57.635258 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:57.635322 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:57.635551 (+ 229us) server_negotiation.cc:299] Negotiation successful
0619 14:50:57.635627 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260619 14:50:57.635795 20077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:57.631748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39681 (local address 127.0.98.1:48113)
0619 14:50:57.632087 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:57.632109 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:57.632269 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:57.632618 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:57.632621 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:57.632660 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:50:57.632803 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:57.632809 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:57.633552 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:57.633559 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:57.634459 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:57.634468 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:57.635287 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:57.635291 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:57.635385 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:57.635412 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 14:50:57.635515 (+ 103us) client_negotiation.cc:240] Negotiation successful
0619 14:50:57.635607 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":122,"threads_started":1}
I20260619 14:50:57.636758 20078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:57.632224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38025 (local address 127.0.98.1:46015)
0619 14:50:57.632530 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:57.632544 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:57.632686 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:57.632931 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:57.632934 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:57.633062 (+ 128us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:50:57.633260 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:57.633271 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:57.634166 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:57.634179 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:57.635502 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:57.635513 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:57.636329 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:57.636333 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:57.636436 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:57.636465 (+ 29us) client_negotiation.cc:769] Sending connection context
0619 14:50:57.636529 (+ 64us) client_negotiation.cc:240] Negotiation successful
0619 14:50:57.636581 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":139,"threads_started":1}
I20260619 14:50:57.636791 20067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:57.632359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46015 (local address 127.0.98.2:38025)
0619 14:50:57.632508 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:57.632513 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:57.632636 (+ 123us) server_negotiation.cc:407] Connection header received
0619 14:50:57.632773 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:57.632776 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:57.632825 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:57.632909 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:50:57.633440 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:57.634053 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:57.635600 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:57.636200 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:57.636295 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:57.636344 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:57.636573 (+ 229us) server_negotiation.cc:299] Negotiation successful
0619 14:50:57.636674 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 14:50:57.636790 20005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da949129e2cc4941bc51b6ad2971f213" candidate_uuid: "ae9bb40322b84d9990b6e4a969e49a59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcea9027465843a18380a2bebe5ac03f" is_pre_election: true
I20260619 14:50:57.637137 20005 raft_consensus.cc:2454] T da949129e2cc4941bc51b6ad2971f213 P bcea9027465843a18380a2bebe5ac03f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9bb40322b84d9990b6e4a969e49a59 in term 0.
I20260619 14:50:57.637635 19671 leader_election.cc:302] T da949129e2cc4941bc51b6ad2971f213 P ae9bb40322b84d9990b6e4a969e49a59 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae9bb40322b84d9990b6e4a969e49a59, bcea9027465843a18380a2bebe5ac03f; no voters:
I20260619 14:50:57.637959 19871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da949129e2cc4941bc51b6ad2971f213" candidate_uuid: "ae9bb40322b84d9990b6e4a969e49a59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5862e1db953a49bca84a316949eaf9bf" is_pre_election: true
I20260619 14:50:57.638027 20076 raft_consensus.cc:2781] T da949129e2cc4941bc51b6ad2971f213 P ae9bb40322b84d9990b6e4a969e49a59 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:50:57.638159 20076 raft_consensus.cc:482] T da949129e2cc4941bc51b6ad2971f213 P ae9bb40322b84d9990b6e4a969e49a59 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:50:57.638216 20076 raft_consensus.cc:3037] T da949129e2cc4941bc51b6ad2971f213 P ae9bb40322b84d9990b6e4a969e49a59 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:50:57.638329 19871 raft_consensus.cc:2454] T da949129e2cc4941bc51b6ad2971f213 P 5862e1db953a49bca84a316949eaf9bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae9bb40322b84d9990b6e4a969e49a59 in term 0.
I20260619 14:50:57.639459 20076 raft_consensus.cc:504] T da949129e2cc4941bc51b6ad2971f213 P ae9bb40322b84d9990b6e4a969e49a59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae9bb40322b84d9990b6e4a969e49a59" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36599 } } peers { permanent_uuid: "bcea9027465843a18380a2bebe5ac03f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39681 } } peers { permanent_uuid: "5862e1db953a49bca84a316949eaf9bf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38025 } }
I20260619 14:50:57.639827 20076 leader_election.cc:288] T da949129e2cc4941bc51b6ad2971f213 P ae9bb40322b84d9990b6e4a969e49a59 [CANDIDATE]: Term 1 election: Requested vote from peers bcea9027465843a18380a2bebe5ac03f (127.0.98.3:39681), 5862e1db953a49bca84a316949eaf9bf (127.0.98.2:38025)
I20260619 14:50:57.640319 20005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da949129e2cc4941bc51b6ad2971f213" candidate_uuid: "ae9bb40322b84d9990b6e4a969e49a59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcea9027465843a18380a2bebe5ac03f"
I20260619 14:50:57.640517 19871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da949129e2cc4941bc51b6ad2971f213" candidate_uuid: "ae9bb40322b84d9990b6e4a969e49a59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5862e1db953a49bca84a316949eaf9bf"
I20260619 14:50:57.640493 20005 raft_consensus.cc:3037] T da949129e2cc4941bc51b6ad2971f213 P bcea9027465843a18380a2bebe5ac03f [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:50:57.640684 19871 raft_consensus.cc:3037] T da949129e2cc4941bc51b6ad2971f213 P 5862e1db953a49bca84a316949eaf9bf [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:50:57.641834 19871 raft_consensus.cc:2454] T da949129e2cc4941bc51b6ad2971f213 P 5862e1db953a49bca84a316949eaf9bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9bb40322b84d9990b6e4a969e49a59 in term 1.
I20260619 14:50:57.642050 20005 raft_consensus.cc:2454] T da949129e2cc4941bc51b6ad2971f213 P bcea9027465843a18380a2bebe5ac03f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae9bb40322b84d9990b6e4a969e49a59 in term 1.
I20260619 14:50:57.642238 19671 leader_election.cc:302] T da949129e2cc4941bc51b6ad2971f213 P ae9bb40322b84d9990b6e4a969e49a59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5862e1db953a49bca84a316949eaf9bf, ae9bb40322b84d9990b6e4a969e49a59; no voters:
I20260619 14:50:57.642457 20076 raft_consensus.cc:2781] T da949129e2cc4941bc51b6ad2971f213 P ae9bb40322b84d9990b6e4a969e49a59 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:50:57.642706 20076 raft_consensus.cc:686] T da949129e2cc4941bc51b6ad2971f213 P ae9bb40322b84d9990b6e4a969e49a59 [term 1 LEADER]: Becoming Leader. State: Replica: ae9bb40322b84d9990b6e4a969e49a59, State: Running, Role: LEADER
I20260619 14:50:57.643110 20076 consensus_queue.cc:237] T da949129e2cc4941bc51b6ad2971f213 P ae9bb40322b84d9990b6e4a969e49a59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae9bb40322b84d9990b6e4a969e49a59" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36599 } } peers { permanent_uuid: "bcea9027465843a18380a2bebe5ac03f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39681 } } peers { permanent_uuid: "5862e1db953a49bca84a316949eaf9bf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38025 } }
I20260619 14:50:57.647078 19590 catalog_manager.cc:5495] T da949129e2cc4941bc51b6ad2971f213 P ae9bb40322b84d9990b6e4a969e49a59 reported cstate change: term changed from 0 to 1, leader changed from <none> to ae9bb40322b84d9990b6e4a969e49a59 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "ae9bb40322b84d9990b6e4a969e49a59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae9bb40322b84d9990b6e4a969e49a59" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36599 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcea9027465843a18380a2bebe5ac03f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39681 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5862e1db953a49bca84a316949eaf9bf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38025 } health_report { overall_health: UNKNOWN } } }
W20260619 14:50:57.682857 19918 tablet.cc:2369] T da949129e2cc4941bc51b6ad2971f213 P 5862e1db953a49bca84a316949eaf9bf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 14:50:57.684893 19784 tablet.cc:2369] T da949129e2cc4941bc51b6ad2971f213 P ae9bb40322b84d9990b6e4a969e49a59: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:50:57.690531 20063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:57.686207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36340 (local address 127.0.98.1:36599)
0619 14:50:57.686396 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:57.686400 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:57.686415 (+ 15us) server_negotiation.cc:407] Connection header received
0619 14:50:57.686514 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:57.686518 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:57.686575 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:57.686671 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:50:57.686926 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:57.687486 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:57.688606 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:57.688962 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:57.689066 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:57.689530 (+ 464us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:50:57.690027 (+ 497us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:50:57.690145 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:57.690329 (+ 184us) server_negotiation.cc:299] Negotiation successful
0619 14:50:57.690400 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
W20260619 14:50:57.690764 20052 tablet.cc:2369] T da949129e2cc4941bc51b6ad2971f213 P bcea9027465843a18380a2bebe5ac03f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:50:57.698081 19871 raft_consensus.cc:1261] T da949129e2cc4941bc51b6ad2971f213 P 5862e1db953a49bca84a316949eaf9bf [term 1 FOLLOWER]: Refusing update from remote peer ae9bb40322b84d9990b6e4a969e49a59: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:50:57.698122 20005 raft_consensus.cc:1261] T da949129e2cc4941bc51b6ad2971f213 P bcea9027465843a18380a2bebe5ac03f [term 1 FOLLOWER]: Refusing update from remote peer ae9bb40322b84d9990b6e4a969e49a59: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:50:57.698952 20076 consensus_queue.cc:1035] T da949129e2cc4941bc51b6ad2971f213 P ae9bb40322b84d9990b6e4a969e49a59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5862e1db953a49bca84a316949eaf9bf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:50:57.699205 20079 consensus_queue.cc:1035] T da949129e2cc4941bc51b6ad2971f213 P ae9bb40322b84d9990b6e4a969e49a59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcea9027465843a18380a2bebe5ac03f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:50:57.709755 20084 mvcc.cc:205] Tried to move back new op lower bound from 7298583173921693696 to 7298583173709365248. Current Snapshot: MvccSnapshot[applied={T|T < 7298583173921693696}]
I20260619 14:50:57.717592 19591 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42132:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:50:57.717835 19591 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42132:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:50:57.720880 19591 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a0cd44655d674b70ad95c298af92d2e9: Sending DeleteTablet for 3 replicas of tablet da949129e2cc4941bc51b6ad2971f213
I20260619 14:50:57.721800 19717 tablet_service.cc:1514] Processing DeleteTablet for tablet da949129e2cc4941bc51b6ad2971f213 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:50:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36326
I20260619 14:50:57.721915 19985 tablet_service.cc:1514] Processing DeleteTablet for tablet da949129e2cc4941bc51b6ad2971f213 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:50:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54060
I20260619 14:50:57.722421 19851 tablet_service.cc:1514] Processing DeleteTablet for tablet da949129e2cc4941bc51b6ad2971f213 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:50:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41012
I20260619 14:50:57.722648 20091 tablet_replica.cc:330] T da949129e2cc4941bc51b6ad2971f213 P ae9bb40322b84d9990b6e4a969e49a59: stopping tablet replica
I20260619 14:50:57.722889 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 19654
I20260619 14:50:57.722963 20092 tablet_replica.cc:330] T da949129e2cc4941bc51b6ad2971f213 P bcea9027465843a18380a2bebe5ac03f: stopping tablet replica
I20260619 14:50:57.723193 20091 raft_consensus.cc:2229] T da949129e2cc4941bc51b6ad2971f213 P ae9bb40322b84d9990b6e4a969e49a59 [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:50:57.723224 20092 raft_consensus.cc:2229] T da949129e2cc4941bc51b6ad2971f213 P bcea9027465843a18380a2bebe5ac03f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:50:57.723574 20092 raft_consensus.cc:2258] T da949129e2cc4941bc51b6ad2971f213 P bcea9027465843a18380a2bebe5ac03f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:50:57.723693 20093 tablet_replica.cc:330] T da949129e2cc4941bc51b6ad2971f213 P 5862e1db953a49bca84a316949eaf9bf: stopping tablet replica
I20260619 14:50:57.724007 20093 raft_consensus.cc:2229] T da949129e2cc4941bc51b6ad2971f213 P 5862e1db953a49bca84a316949eaf9bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:50:57.724321 20093 raft_consensus.cc:2258] T da949129e2cc4941bc51b6ad2971f213 P 5862e1db953a49bca84a316949eaf9bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:50:57.725404 20092 ts_tablet_manager.cc:1905] T da949129e2cc4941bc51b6ad2971f213 P bcea9027465843a18380a2bebe5ac03f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:50:57.729161 20092 ts_tablet_manager.cc:1918] T da949129e2cc4941bc51b6ad2971f213 P bcea9027465843a18380a2bebe5ac03f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:50:57.729285 20092 log.cc:1196] T da949129e2cc4941bc51b6ad2971f213 P bcea9027465843a18380a2bebe5ac03f: Deleting WAL directory at /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781880383132139-392-0/minicluster-data/ts-2/wal/wals/da949129e2cc4941bc51b6ad2971f213
I20260619 14:50:57.729655 20092 ts_tablet_manager.cc:1939] T da949129e2cc4941bc51b6ad2971f213 P bcea9027465843a18380a2bebe5ac03f: Deleting consensus metadata
I20260619 14:50:57.730154 20093 ts_tablet_manager.cc:1905] T da949129e2cc4941bc51b6ad2971f213 P 5862e1db953a49bca84a316949eaf9bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:50:57.731244 19577 catalog_manager.cc:4841] TS bcea9027465843a18380a2bebe5ac03f (127.0.98.3:39681): tablet da949129e2cc4941bc51b6ad2971f213 (table test-table [id=1ce95aab4abe4d2fbcc99cd701ca9429]) successfully deleted
W20260619 14:50:57.731767 19578 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36599 (error 108)
I20260619 14:50:57.732781 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 19788
I20260619 14:50:57.732988 20066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:57.732641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36599 (local address 127.0.0.1:36354)
0619 14:50:57.732800 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:57.732883 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260619 14:50:57.737370 19578 catalog_manager.cc:4584] TS ae9bb40322b84d9990b6e4a969e49a59 (127.0.98.1:36599): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet da949129e2cc4941bc51b6ad2971f213: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36599: connect: Connection refused (error 111)
I20260619 14:50:57.739277 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 19922
I20260619 14:50:57.740773 20066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:57.739456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38025 (local address 127.0.0.1:41020)
0619 14:50:57.740558 (+ 1102us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:57.740636 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:38025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":837}
I20260619 14:50:57.746023 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 19560
2026-06-19T14:50:57Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2593 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-19T14:50:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:50:57Z Disabled control of system clock
I20260619 14:50:57.779560 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880383132139-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880383132139-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880383132139-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40379
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:45291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40379
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 14:50:57.899279 20098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:50:57.899595 20098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:50:57.899634 20098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:50:57.903815 20098 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:50:57.903904 20098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:50:57.903919 20098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:50:57.903936 20098 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:50:57.903952 20098 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:50:57.908968 20098 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880383132139-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880383132139-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40379
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20098
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880383132139-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:50:57.910075 20098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:50:57.911453 20098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:50:57.920603 20103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:57.920559 20106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:57.920619 20104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:50:57.920713 20098 server_base.cc:1034] running on GCE node
I20260619 14:50:57.921627 20098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:50:57.922853 20098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:50:57.924062 20098 hybrid_clock.cc:648] HybridClock initialized: now 1781880657924040 us; error 37 us; skew 500 ppm
I20260619 14:50:57.933081 20098 webserver.cc:460] Webserver started at http://127.0.98.62:35619/ using document root <none> and password file <none>
I20260619 14:50:57.933771 20098 fs_manager.cc:359] Metadata directory not provided
I20260619 14:50:57.933852 20098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:50:57.934110 20098 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:50:57.936507 20098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880383132139-392-0/minicluster-data/master-0/data/instance:
uuid: "6be54523d86d478db9467c48383f7ecd"
format_stamp: "Formatted at 2026-06-19 14:50:57 on dist-test-slave-2sb1"
server_key: "7e3d6f182ed07623c8b52f66310fa86c"
server_key_iv: "42f31c3dab78eeedfaa4bcce9dc3c316"
server_key_version: "encryptionkey@0"
I20260619 14:50:57.937244 20098 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880383132139-392-0/minicluster-data/master-0/wal/instance:
uuid: "6be54523d86d478db9467c48383f7ecd"
format_stamp: "Formatted at 2026-06-19 14:50:57 on dist-test-slave-2sb1"
server_key: "7e3d6f182ed07623c8b52f66310fa86c"
server_key_iv: "42f31c3dab78eeedfaa4bcce9dc3c316"
server_key_version: "encryptionkey@0"
I20260619 14:50:57.941313 20098 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 14:50:57.944192 20112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:57.945701 20098 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:50:57.945886 20098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880383132139-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880383132139-392-0/minicluster-data/master-0/wal
uuid: "6be54523d86d478db9467c48383f7ecd"
format_stamp: "Formatted at 2026-06-19 14:50:57 on dist-test-slave-2sb1"
server_key: "7e3d6f182ed07623c8b52f66310fa86c"
server_key_iv: "42f31c3dab78eeedfaa4bcce9dc3c316"
server_key_version: "encryptionkey@0"
I20260619 14:50:57.946045 20098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880383132139-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880383132139-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880383132139-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:50:57.981179 20098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:50:57.981886 20098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:50:57.982048 20098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:50:57.989567 20098 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40379
I20260619 14:50:57.989604 20164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40379 every 8 connection(s)
I20260619 14:50:57.990643 20098 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:50:57.994055 20165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:50:57.997136 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 20098
I20260619 14:50:57.997282 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880383132139-392-0/minicluster-data/master-0/wal/instance
I20260619 14:50:57.997563 392 external_mini_cluster.cc:1428] Setting key 5417453204fa5c09e29f054c1b258246
I20260619 14:50:58.000839 20165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6be54523d86d478db9467c48383f7ecd: Bootstrap starting.
I20260619 14:50:58.003077 20165 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6be54523d86d478db9467c48383f7ecd: Neither blocks nor log segments found. Creating new log.
I20260619 14:50:58.003850 20165 log.cc:824] T 00000000000000000000000000000000 P 6be54523d86d478db9467c48383f7ecd: Log is configured to *not* fsync() on all Append() calls
I20260619 14:50:58.005897 20165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6be54523d86d478db9467c48383f7ecd: No bootstrap required, opened a new log
I20260619 14:50:58.007264 20168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:57.999090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44900 (local address 127.0.98.62:40379)
0619 14:50:57.999796 (+ 706us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:57.999811 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:57.999854 (+ 43us) server_negotiation.cc:407] Connection header received
0619 14:50:58.000498 (+ 644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:58.000515 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:58.001077 (+ 562us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:58.001422 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:58.001899 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:58.002869 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:58.003843 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:58.004382 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:58.004564 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:58.005509 (+ 945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:58.005515 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:58.005529 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:50:58.005945 (+ 416us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:58.006032 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:50:58.006059 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:58.006152 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:58.006267 (+ 115us) server_negotiation.cc:299] Negotiation successful
0619 14:50:58.006547 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":140,"threads_started":1}
I20260619 14:50:58.011015 20165 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6be54523d86d478db9467c48383f7ecd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6be54523d86d478db9467c48383f7ecd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40379 } }
I20260619 14:50:58.011436 20165 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6be54523d86d478db9467c48383f7ecd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:50:58.011524 20165 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6be54523d86d478db9467c48383f7ecd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6be54523d86d478db9467c48383f7ecd, State: Initialized, Role: FOLLOWER
I20260619 14:50:58.012040 20165 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6be54523d86d478db9467c48383f7ecd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6be54523d86d478db9467c48383f7ecd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40379 } }
I20260619 14:50:58.012210 20165 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6be54523d86d478db9467c48383f7ecd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:50:58.012310 20165 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6be54523d86d478db9467c48383f7ecd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:50:58.012418 20165 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6be54523d86d478db9467c48383f7ecd [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:50:58.013523 20165 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6be54523d86d478db9467c48383f7ecd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6be54523d86d478db9467c48383f7ecd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40379 } }
I20260619 14:50:58.013906 20165 leader_election.cc:302] T 00000000000000000000000000000000 P 6be54523d86d478db9467c48383f7ecd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6be54523d86d478db9467c48383f7ecd; no voters:
I20260619 14:50:58.014317 20165 leader_election.cc:288] T 00000000000000000000000000000000 P 6be54523d86d478db9467c48383f7ecd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:50:58.014598 20170 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6be54523d86d478db9467c48383f7ecd [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:50:58.015107 20170 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6be54523d86d478db9467c48383f7ecd [term 1 LEADER]: Becoming Leader. State: Replica: 6be54523d86d478db9467c48383f7ecd, State: Running, Role: LEADER
I20260619 14:50:58.015465 20170 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6be54523d86d478db9467c48383f7ecd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6be54523d86d478db9467c48383f7ecd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40379 } }
I20260619 14:50:58.016409 20165 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6be54523d86d478db9467c48383f7ecd [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:50:58.017426 20171 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6be54523d86d478db9467c48383f7ecd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6be54523d86d478db9467c48383f7ecd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6be54523d86d478db9467c48383f7ecd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40379 } } }
I20260619 14:50:58.017566 20171 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6be54523d86d478db9467c48383f7ecd [sys.catalog]: This master's current role is: LEADER
I20260619 14:50:58.017397 20172 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6be54523d86d478db9467c48383f7ecd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6be54523d86d478db9467c48383f7ecd. Latest consensus state: current_term: 1 leader_uuid: "6be54523d86d478db9467c48383f7ecd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6be54523d86d478db9467c48383f7ecd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40379 } } }
I20260619 14:50:58.017647 20172 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6be54523d86d478db9467c48383f7ecd [sys.catalog]: This master's current role is: LEADER
I20260619 14:50:58.017973 20179 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:50:58.021327 20179 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:50:58.027910 20179 catalog_manager.cc:1347] Generated new cluster ID: 34640ce52949488bb76424054ab02e98
I20260619 14:50:58.028023 20179 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:50:58.038051 20179 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:50:58.039242 20179 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:50:58.048750 20179 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6be54523d86d478db9467c48383f7ecd: Generated new TSK 0
I20260619 14:50:58.049701 20179 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:50:58.053484 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880383132139-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880383132139-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880383132139-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880383132139-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40379
--builtin_ntp_servers=127.0.98.20:45291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260619 14:50:58.181332 20189 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:50:58.181617 20189 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:50:58.181658 20189 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:50:58.185918 20189 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:50:58.185997 20189 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:50:58.186056 20189 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260619 14:50:58.191267 20189 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880383132139-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880383132139-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880383132139-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20189
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880383132139-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:50:58.192420 20189 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:50:58.193890 20189 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 14:50:58.203998 20189 server_base.cc:1034] running on GCE node
W20260619 14:50:58.203944 20197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:58.203971 20194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:58.203944 20195 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:50:58.204604 20189 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:50:58.205708 20189 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:50:58.206961 20189 hybrid_clock.cc:648] HybridClock initialized: now 1781880658206945 us; error 32 us; skew 500 ppm
I20260619 14:50:58.209385 20189 webserver.cc:460] Webserver started at http://127.0.98.1:40547/ using document root <none> and password file <none>
I20260619 14:50:58.210026 20189 fs_manager.cc:359] Metadata directory not provided
I20260619 14:50:58.210130 20189 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:50:58.210382 20189 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:50:58.212613 20189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880383132139-392-0/minicluster-data/ts-0/data/instance:
uuid: "c613911f21464cdbbcab16b98107d740"
format_stamp: "Formatted at 2026-06-19 14:50:58 on dist-test-slave-2sb1"
server_key: "78f931b4070138ea91f375c73c3fd75d"
server_key_iv: "dd6d65f78d224aadd712a53353ee148e"
server_key_version: "encryptionkey@0"
I20260619 14:50:58.213269 20189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880383132139-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c613911f21464cdbbcab16b98107d740"
format_stamp: "Formatted at 2026-06-19 14:50:58 on dist-test-slave-2sb1"
server_key: "78f931b4070138ea91f375c73c3fd75d"
server_key_iv: "dd6d65f78d224aadd712a53353ee148e"
server_key_version: "encryptionkey@0"
I20260619 14:50:58.217396 20189 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 14:50:58.220146 20203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:58.221609 20189 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:50:58.221758 20189 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880383132139-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880383132139-392-0/minicluster-data/ts-0/wal
uuid: "c613911f21464cdbbcab16b98107d740"
format_stamp: "Formatted at 2026-06-19 14:50:58 on dist-test-slave-2sb1"
server_key: "78f931b4070138ea91f375c73c3fd75d"
server_key_iv: "dd6d65f78d224aadd712a53353ee148e"
server_key_version: "encryptionkey@0"
I20260619 14:50:58.221930 20189 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880383132139-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880383132139-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880383132139-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:50:58.243119 20189 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:50:58.243988 20189 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:50:58.244227 20189 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:50:58.244993 20189 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:50:58.246075 20189 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:50:58.246155 20189 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:58.246236 20189 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:50:58.246291 20189 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:58.257257 20189 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43033
I20260619 14:50:58.257290 20316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43033 every 8 connection(s)
I20260619 14:50:58.258280 20189 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880383132139-392-0/minicluster-data/ts-0/data/info.pb
I20260619 14:50:58.261718 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 20189
I20260619 14:50:58.261847 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880383132139-392-0/minicluster-data/ts-0/wal/instance
I20260619 14:50:58.262187 392 external_mini_cluster.cc:1428] Setting key 52d31b9e2d2b12c0bbd95fed1615fd77
I20260619 14:50:58.264582 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880383132139-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880383132139-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880383132139-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880383132139-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40379
--builtin_ntp_servers=127.0.98.20:45291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 14:50:58.270882 20168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:58.260445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52271 (local address 127.0.98.62:40379)
0619 14:50:58.260607 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:58.260612 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:58.262279 (+ 1667us) server_negotiation.cc:407] Connection header received
0619 14:50:58.264020 (+ 1741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:58.264023 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:58.264070 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:58.264156 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:58.265627 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:58.266711 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:58.268203 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:58.268611 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:58.268812 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:58.270308 (+ 1496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:58.270311 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:58.270313 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:50:58.270327 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:58.270348 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:50:58.270356 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:58.270421 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:58.270697 (+ 276us) server_negotiation.cc:299] Negotiation successful
0619 14:50:58.270759 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260619 14:50:58.272379 20319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:58.261028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40379 (local address 127.0.98.1:52271)
0619 14:50:58.261993 (+ 965us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:58.262066 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:58.263728 (+ 1662us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:58.264392 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:58.264405 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:58.264529 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:58.265302 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:58.265324 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:58.266944 (+ 1620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:58.266947 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:58.268056 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:58.268068 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:58.268899 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:58.268907 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:58.269184 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:58.269936 (+ 752us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:50:58.269941 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:58.270011 (+ 70us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:50:58.270117 (+ 106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:50:58.270457 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:50:58.270470 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:50:58.270481 (+ 11us) client_negotiation.cc:769] Sending connection context
0619 14:50:58.270650 (+ 169us) client_negotiation.cc:240] Negotiation successful
0619 14:50:58.271053 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":122,"threads_started":1}
I20260619 14:50:58.274084 20317 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40379
I20260619 14:50:58.274379 20317 heartbeater.cc:464] Registering TS with master...
I20260619 14:50:58.274964 20317 heartbeater.cc:511] Master 127.0.98.62:40379 requested a full tablet report, sending...
I20260619 14:50:58.276868 20129 ts_manager.cc:194] Registered new tserver with Master: c613911f21464cdbbcab16b98107d740 (127.0.98.1:43033)
I20260619 14:50:58.278052 20129 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:52271
W20260619 14:50:58.397442 20320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:50:58.397817 20320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:50:58.397900 20320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:50:58.402732 20320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:50:58.402849 20320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:50:58.402974 20320 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260619 14:50:58.409080 20320 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880383132139-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880383132139-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880383132139-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20320
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880383132139-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:50:58.410373 20320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:50:58.411890 20320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:50:58.423198 20325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:58.423221 20328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:58.423223 20326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:50:58.423481 20320 server_base.cc:1034] running on GCE node
I20260619 14:50:58.424077 20320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:50:58.424816 20320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:50:58.425997 20320 hybrid_clock.cc:648] HybridClock initialized: now 1781880658425939 us; error 75 us; skew 500 ppm
I20260619 14:50:58.427966 20320 webserver.cc:460] Webserver started at http://127.0.98.2:39795/ using document root <none> and password file <none>
I20260619 14:50:58.428561 20320 fs_manager.cc:359] Metadata directory not provided
I20260619 14:50:58.428627 20320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:50:58.428893 20320 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:50:58.431075 20320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880383132139-392-0/minicluster-data/ts-1/data/instance:
uuid: "73d5892734b54ffab87341c35af43c74"
format_stamp: "Formatted at 2026-06-19 14:50:58 on dist-test-slave-2sb1"
server_key: "c75d2230cddeb5b0c4b35ca052dcce40"
server_key_iv: "94b196f4008414f2635243f924e92f9f"
server_key_version: "encryptionkey@0"
I20260619 14:50:58.431674 20320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880383132139-392-0/minicluster-data/ts-1/wal/instance:
uuid: "73d5892734b54ffab87341c35af43c74"
format_stamp: "Formatted at 2026-06-19 14:50:58 on dist-test-slave-2sb1"
server_key: "c75d2230cddeb5b0c4b35ca052dcce40"
server_key_iv: "94b196f4008414f2635243f924e92f9f"
server_key_version: "encryptionkey@0"
I20260619 14:50:58.436293 20320 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260619 14:50:58.439211 20334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:58.440902 20320 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:50:58.441062 20320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880383132139-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880383132139-392-0/minicluster-data/ts-1/wal
uuid: "73d5892734b54ffab87341c35af43c74"
format_stamp: "Formatted at 2026-06-19 14:50:58 on dist-test-slave-2sb1"
server_key: "c75d2230cddeb5b0c4b35ca052dcce40"
server_key_iv: "94b196f4008414f2635243f924e92f9f"
server_key_version: "encryptionkey@0"
I20260619 14:50:58.441195 20320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880383132139-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880383132139-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880383132139-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:50:58.455717 20320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:50:58.456523 20320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:50:58.456775 20320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:50:58.457420 20320 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:50:58.458472 20320 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:50:58.458559 20320 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:58.458634 20320 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:50:58.458678 20320 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:58.468549 20320 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44161
I20260619 14:50:58.468565 20447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44161 every 8 connection(s)
I20260619 14:50:58.469552 20320 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880383132139-392-0/minicluster-data/ts-1/data/info.pb
I20260619 14:50:58.474488 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 20320
I20260619 14:50:58.474596 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880383132139-392-0/minicluster-data/ts-1/wal/instance
I20260619 14:50:58.474952 392 external_mini_cluster.cc:1428] Setting key ed77081ae7f49f9aee99768a78f6e46a
I20260619 14:50:58.477280 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880383132139-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880383132139-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880383132139-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880383132139-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40379
--builtin_ntp_servers=127.0.98.20:45291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260619 14:50:58.481678 20168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:58.471725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38583 (local address 127.0.98.62:40379)
0619 14:50:58.471905 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:58.471909 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:58.473286 (+ 1377us) server_negotiation.cc:407] Connection header received
0619 14:50:58.474906 (+ 1620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:58.474912 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:58.474977 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:58.475091 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:58.476246 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:58.476966 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:58.478868 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:58.479465 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:58.479622 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:58.481094 (+ 1472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:58.481097 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:58.481099 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:50:58.481114 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:58.481135 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:50:58.481144 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:58.481220 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:58.481480 (+ 260us) server_negotiation.cc:299] Negotiation successful
0619 14:50:58.481548 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260619 14:50:58.483106 20450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:58.472169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40379 (local address 127.0.98.2:38583)
0619 14:50:58.473088 (+ 919us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:58.473132 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:58.474612 (+ 1480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:58.475289 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:58.475297 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:58.475392 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:58.476095 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:58.476111 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:58.477140 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:58.477145 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:58.478690 (+ 1545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:58.478706 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:58.479692 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:58.479695 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:58.479867 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:58.480680 (+ 813us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:50:58.480685 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:58.480768 (+ 83us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:50:58.480874 (+ 106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:50:58.481250 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:50:58.481264 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:50:58.481275 (+ 11us) client_negotiation.cc:769] Sending connection context
0619 14:50:58.481438 (+ 163us) client_negotiation.cc:240] Negotiation successful
0619 14:50:58.481811 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":165,"threads_started":1}
I20260619 14:50:58.484845 20448 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40379
I20260619 14:50:58.485136 20448 heartbeater.cc:464] Registering TS with master...
I20260619 14:50:58.485797 20448 heartbeater.cc:511] Master 127.0.98.62:40379 requested a full tablet report, sending...
I20260619 14:50:58.487092 20129 ts_manager.cc:194] Registered new tserver with Master: 73d5892734b54ffab87341c35af43c74 (127.0.98.2:44161)
I20260619 14:50:58.488108 20129 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:38583
W20260619 14:50:58.606130 20451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:50:58.606441 20451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:50:58.606483 20451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:50:58.610885 20451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:50:58.610975 20451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:50:58.611039 20451 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260619 14:50:58.616209 20451 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880383132139-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880383132139-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880383132139-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20451
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880383132139-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:50:58.617547 20451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:50:58.619547 20451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:50:58.631095 20456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:58.631827 20459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:50:58.631829 20457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:50:58.631918 20451 server_base.cc:1034] running on GCE node
I20260619 14:50:58.632503 20451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:50:58.633256 20451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:50:58.634469 20451 hybrid_clock.cc:648] HybridClock initialized: now 1781880658634438 us; error 46 us; skew 500 ppm
I20260619 14:50:58.636591 20451 webserver.cc:460] Webserver started at http://127.0.98.3:40665/ using document root <none> and password file <none>
I20260619 14:50:58.637239 20451 fs_manager.cc:359] Metadata directory not provided
I20260619 14:50:58.637341 20451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:50:58.637549 20451 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:50:58.639773 20451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880383132139-392-0/minicluster-data/ts-2/data/instance:
uuid: "97e0feafcb89472bb177e25768077b1b"
format_stamp: "Formatted at 2026-06-19 14:50:58 on dist-test-slave-2sb1"
server_key: "be5ca6d2a68e4f161ecbc7917a4e5064"
server_key_iv: "fe4345734093f320944c87b3b3b13f98"
server_key_version: "encryptionkey@0"
I20260619 14:50:58.640426 20451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880383132139-392-0/minicluster-data/ts-2/wal/instance:
uuid: "97e0feafcb89472bb177e25768077b1b"
format_stamp: "Formatted at 2026-06-19 14:50:58 on dist-test-slave-2sb1"
server_key: "be5ca6d2a68e4f161ecbc7917a4e5064"
server_key_iv: "fe4345734093f320944c87b3b3b13f98"
server_key_version: "encryptionkey@0"
I20260619 14:50:58.645109 20451 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 14:50:58.648214 20465 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:58.649868 20451 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:50:58.650039 20451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880383132139-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880383132139-392-0/minicluster-data/ts-2/wal
uuid: "97e0feafcb89472bb177e25768077b1b"
format_stamp: "Formatted at 2026-06-19 14:50:58 on dist-test-slave-2sb1"
server_key: "be5ca6d2a68e4f161ecbc7917a4e5064"
server_key_iv: "fe4345734093f320944c87b3b3b13f98"
server_key_version: "encryptionkey@0"
I20260619 14:50:58.650184 20451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880383132139-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880383132139-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880383132139-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:50:58.678157 20451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:50:58.678933 20451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:50:58.679148 20451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:50:58.679769 20451 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260619 14:50:58.680975 20451 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260619 14:50:58.681046 20451 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:58.681113 20451 ts_tablet_manager.cc:610] Registered 0 tablets
I20260619 14:50:58.681159 20451 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260619 14:50:58.690778 20451 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41559
I20260619 14:50:58.690804 20578 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41559 every 8 connection(s)
I20260619 14:50:58.691766 20451 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880383132139-392-0/minicluster-data/ts-2/data/info.pb
I20260619 14:50:58.697021 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 20451
I20260619 14:50:58.697160 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880383132139-392-0/minicluster-data/ts-2/wal/instance
I20260619 14:50:58.697628 392 external_mini_cluster.cc:1428] Setting key 94768cf88ca4653c34e1edbb50647a4e
I20260619 14:50:58.702872 20168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:58.693874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52483 (local address 127.0.98.62:40379)
0619 14:50:58.694034 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:58.694039 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:58.695260 (+ 1221us) server_negotiation.cc:407] Connection header received
0619 14:50:58.696826 (+ 1566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:58.696829 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:58.696880 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:58.696962 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:58.698248 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:58.699085 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:58.700755 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:58.701021 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:58.701113 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:58.702427 (+ 1314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:58.702430 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:58.702432 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:50:58.702445 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:58.702465 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:50:58.702473 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:58.702537 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:58.702686 (+ 149us) server_negotiation.cc:299] Negotiation successful
0619 14:50:58.702752 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 14:50:58.704001 20581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:58.694304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40379 (local address 127.0.98.3:52483)
0619 14:50:58.695065 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:58.695112 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:58.696558 (+ 1446us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:58.697171 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:58.697185 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:58.697305 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:50:58.698063 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:58.698082 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:58.699235 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:58.699239 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:58.700563 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:58.700575 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:58.701231 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:58.701235 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:58.701461 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:58.702138 (+ 677us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0619 14:50:58.702140 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:50:58.702189 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0619 14:50:58.702265 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:50:58.702558 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:50:58.702565 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:50:58.702569 (+ 4us) client_negotiation.cc:769] Sending connection context
0619 14:50:58.702675 (+ 106us) client_negotiation.cc:240] Negotiation successful
0619 14:50:58.702943 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":118,"threads_started":1}
I20260619 14:50:58.705518 20579 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40379
I20260619 14:50:58.705757 20579 heartbeater.cc:464] Registering TS with master...
I20260619 14:50:58.706270 20579 heartbeater.cc:511] Master 127.0.98.62:40379 requested a full tablet report, sending...
I20260619 14:50:58.707515 20129 ts_manager.cc:194] Registered new tserver with Master: 97e0feafcb89472bb177e25768077b1b (127.0.98.3:41559)
I20260619 14:50:58.708204 20129 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:52483
I20260619 14:50:58.712468 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260619 14:50:58.720264 20168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:58.714729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44916 (local address 127.0.98.62:40379)
0619 14:50:58.714909 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:58.714915 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:58.715057 (+ 142us) server_negotiation.cc:407] Connection header received
0619 14:50:58.715145 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:58.715148 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:58.715193 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:58.715277 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:50:58.715677 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:58.716263 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:58.717548 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:58.717905 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:58.717979 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:58.719612 (+ 1633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:50:58.719620 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:50:58.719623 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:50:58.719642 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:50:58.719671 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:50:58.719683 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:50:58.719777 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:58.719996 (+ 219us) server_negotiation.cc:299] Negotiation successful
0619 14:50:58.720060 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260619 14:50:58.724491 20129 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44916:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260619 14:50:58.727813 20129 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260619 14:50:58.744278 20590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:58.737832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43033 (local address 127.0.0.1:52430)
0619 14:50:58.738517 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:58.738573 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:58.738858 (+ 285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:58.739952 (+ 1094us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:58.739955 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:58.740000 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:50:58.740096 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:58.740101 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:58.741711 (+ 1610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:58.741714 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:58.742820 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:58.742830 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:58.743689 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:58.743693 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:58.743803 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:58.743840 (+ 37us) client_negotiation.cc:769] Sending connection context
0619 14:50:58.743949 (+ 109us) client_negotiation.cc:240] Negotiation successful
0619 14:50:58.744080 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"mutex_wait_us":95,"spinlock_wait_cycles":3840,"thread_start_us":119,"threads_started":1}
I20260619 14:50:58.744275 20592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:58.737981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52430 (local address 127.0.98.1:43033)
0619 14:50:58.739476 (+ 1495us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:58.739483 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:58.739509 (+ 26us) server_negotiation.cc:407] Connection header received
0619 14:50:58.739586 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:58.739590 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:58.739779 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:58.740243 (+ 464us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:50:58.740392 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:58.741571 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:58.742953 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:58.743554 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:58.743687 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:58.743912 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:58.743975 (+ 63us) server_negotiation.cc:299] Negotiation successful
0619 14:50:58.744087 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1384,"thread_start_us":111,"threads_started":1}
I20260619 14:50:58.744335 20593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:58.738147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41559 (local address 127.0.0.1:37762)
0619 14:50:58.739125 (+ 978us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:58.739138 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:58.739223 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:58.739734 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:58.739740 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:58.739764 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:50:58.739929 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:58.739939 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:58.741204 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:58.741209 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:58.742584 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:58.742594 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:58.743818 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:58.743823 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:58.743978 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:58.744003 (+ 25us) client_negotiation.cc:769] Sending connection context
0619 14:50:58.744076 (+ 73us) client_negotiation.cc:240] Negotiation successful
0619 14:50:58.744158 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":921,"mutex_wait_us":17,"thread_start_us":108,"threads_started":1}
I20260619 14:50:58.744551 20595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:58.738691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37762 (local address 127.0.98.3:41559)
0619 14:50:58.739241 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:58.739254 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:58.739291 (+ 37us) server_negotiation.cc:407] Connection header received
0619 14:50:58.739360 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:58.739364 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:58.739560 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:58.739736 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:50:58.740070 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:58.741072 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:58.742728 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:58.743669 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:58.743948 (+ 279us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:58.744173 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:58.744229 (+ 56us) server_negotiation.cc:299] Negotiation successful
0619 14:50:58.744328 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":140,"threads_started":1}
I20260619 14:50:58.746860 20591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:58.737884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44161 (local address 127.0.0.1:35342)
0619 14:50:58.738779 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:58.738794 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:58.738867 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:58.739956 (+ 1089us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:58.739958 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:58.740017 (+ 59us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:50:58.740148 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:58.740155 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:58.741711 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:58.741714 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:58.744842 (+ 3128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:58.744854 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:58.746356 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:58.746359 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:58.746469 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:58.746496 (+ 27us) client_negotiation.cc:769] Sending connection context
0619 14:50:58.746585 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 14:50:58.746692 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":820,"mutex_wait_us":106,"spinlock_wait_cycles":2432,"thread_start_us":130,"threads_started":1}
I20260619 14:50:58.746912 20594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:58.738688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35342 (local address 127.0.98.2:44161)
0619 14:50:58.739196 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:58.739210 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:58.739236 (+ 26us) server_negotiation.cc:407] Connection header received
0619 14:50:58.739327 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:58.739333 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:58.739561 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:58.739736 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:50:58.740285 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:58.741583 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:58.745019 (+ 3436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:58.745964 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:58.746093 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:58.746282 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:58.746638 (+ 356us) server_negotiation.cc:299] Negotiation successful
0619 14:50:58.746747 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":103,"threads_started":1}
I20260619 14:50:58.748072 20513 tablet_service.cc:1467] Processing CreateTablet for tablet 26d8dbba33a54ad49662054e893933a4 (DEFAULT_TABLE table=test-table [id=8956e742fa2948299420f98e939c1e87]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:50:58.747998 20251 tablet_service.cc:1467] Processing CreateTablet for tablet 26d8dbba33a54ad49662054e893933a4 (DEFAULT_TABLE table=test-table [id=8956e742fa2948299420f98e939c1e87]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:50:58.749194 20251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26d8dbba33a54ad49662054e893933a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:50:58.749259 20513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26d8dbba33a54ad49662054e893933a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:50:58.750247 20382 tablet_service.cc:1467] Processing CreateTablet for tablet 26d8dbba33a54ad49662054e893933a4 (DEFAULT_TABLE table=test-table [id=8956e742fa2948299420f98e939c1e87]), partition=RANGE (key) PARTITION UNBOUNDED
I20260619 14:50:58.751369 20382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26d8dbba33a54ad49662054e893933a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:50:58.757400 20597 tablet_bootstrap.cc:492] T 26d8dbba33a54ad49662054e893933a4 P c613911f21464cdbbcab16b98107d740: Bootstrap starting.
I20260619 14:50:58.758345 20596 tablet_bootstrap.cc:492] T 26d8dbba33a54ad49662054e893933a4 P 97e0feafcb89472bb177e25768077b1b: Bootstrap starting.
I20260619 14:50:58.759019 20598 tablet_bootstrap.cc:492] T 26d8dbba33a54ad49662054e893933a4 P 73d5892734b54ffab87341c35af43c74: Bootstrap starting.
I20260619 14:50:58.760210 20597 tablet_bootstrap.cc:654] T 26d8dbba33a54ad49662054e893933a4 P c613911f21464cdbbcab16b98107d740: Neither blocks nor log segments found. Creating new log.
I20260619 14:50:58.760612 20596 tablet_bootstrap.cc:654] T 26d8dbba33a54ad49662054e893933a4 P 97e0feafcb89472bb177e25768077b1b: Neither blocks nor log segments found. Creating new log.
I20260619 14:50:58.760985 20598 tablet_bootstrap.cc:654] T 26d8dbba33a54ad49662054e893933a4 P 73d5892734b54ffab87341c35af43c74: Neither blocks nor log segments found. Creating new log.
I20260619 14:50:58.761286 20597 log.cc:824] T 26d8dbba33a54ad49662054e893933a4 P c613911f21464cdbbcab16b98107d740: Log is configured to *not* fsync() on all Append() calls
I20260619 14:50:58.761612 20596 log.cc:824] T 26d8dbba33a54ad49662054e893933a4 P 97e0feafcb89472bb177e25768077b1b: Log is configured to *not* fsync() on all Append() calls
I20260619 14:50:58.761765 20598 log.cc:824] T 26d8dbba33a54ad49662054e893933a4 P 73d5892734b54ffab87341c35af43c74: Log is configured to *not* fsync() on all Append() calls
I20260619 14:50:58.763693 20598 tablet_bootstrap.cc:492] T 26d8dbba33a54ad49662054e893933a4 P 73d5892734b54ffab87341c35af43c74: No bootstrap required, opened a new log
I20260619 14:50:58.763927 20598 ts_tablet_manager.cc:1397] T 26d8dbba33a54ad49662054e893933a4 P 73d5892734b54ffab87341c35af43c74: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260619 14:50:58.764072 20597 tablet_bootstrap.cc:492] T 26d8dbba33a54ad49662054e893933a4 P c613911f21464cdbbcab16b98107d740: No bootstrap required, opened a new log
I20260619 14:50:58.764236 20597 ts_tablet_manager.cc:1397] T 26d8dbba33a54ad49662054e893933a4 P c613911f21464cdbbcab16b98107d740: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260619 14:50:58.766605 20596 tablet_bootstrap.cc:492] T 26d8dbba33a54ad49662054e893933a4 P 97e0feafcb89472bb177e25768077b1b: No bootstrap required, opened a new log
I20260619 14:50:58.766861 20596 ts_tablet_manager.cc:1397] T 26d8dbba33a54ad49662054e893933a4 P 97e0feafcb89472bb177e25768077b1b: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260619 14:50:58.767735 20597 raft_consensus.cc:348] T 26d8dbba33a54ad49662054e893933a4 P c613911f21464cdbbcab16b98107d740 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c613911f21464cdbbcab16b98107d740" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43033 } } peers { permanent_uuid: "73d5892734b54ffab87341c35af43c74" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44161 } } peers { permanent_uuid: "97e0feafcb89472bb177e25768077b1b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41559 } }
I20260619 14:50:58.767972 20597 raft_consensus.cc:374] T 26d8dbba33a54ad49662054e893933a4 P c613911f21464cdbbcab16b98107d740 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:50:58.768013 20597 raft_consensus.cc:729] T 26d8dbba33a54ad49662054e893933a4 P c613911f21464cdbbcab16b98107d740 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c613911f21464cdbbcab16b98107d740, State: Initialized, Role: FOLLOWER
I20260619 14:50:58.768520 20597 consensus_queue.cc:260] T 26d8dbba33a54ad49662054e893933a4 P c613911f21464cdbbcab16b98107d740 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c613911f21464cdbbcab16b98107d740" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43033 } } peers { permanent_uuid: "73d5892734b54ffab87341c35af43c74" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44161 } } peers { permanent_uuid: "97e0feafcb89472bb177e25768077b1b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41559 } }
I20260619 14:50:58.769323 20317 heartbeater.cc:503] Master 127.0.98.62:40379 was elected leader, sending a full tablet report...
I20260619 14:50:58.769327 20598 raft_consensus.cc:348] T 26d8dbba33a54ad49662054e893933a4 P 73d5892734b54ffab87341c35af43c74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c613911f21464cdbbcab16b98107d740" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43033 } } peers { permanent_uuid: "73d5892734b54ffab87341c35af43c74" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44161 } } peers { permanent_uuid: "97e0feafcb89472bb177e25768077b1b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41559 } }
I20260619 14:50:58.769665 20598 raft_consensus.cc:374] T 26d8dbba33a54ad49662054e893933a4 P 73d5892734b54ffab87341c35af43c74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:50:58.769723 20598 raft_consensus.cc:729] T 26d8dbba33a54ad49662054e893933a4 P 73d5892734b54ffab87341c35af43c74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73d5892734b54ffab87341c35af43c74, State: Initialized, Role: FOLLOWER
I20260619 14:50:58.770319 20598 consensus_queue.cc:260] T 26d8dbba33a54ad49662054e893933a4 P 73d5892734b54ffab87341c35af43c74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c613911f21464cdbbcab16b98107d740" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43033 } } peers { permanent_uuid: "73d5892734b54ffab87341c35af43c74" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44161 } } peers { permanent_uuid: "97e0feafcb89472bb177e25768077b1b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41559 } }
I20260619 14:50:58.770752 20597 ts_tablet_manager.cc:1428] T 26d8dbba33a54ad49662054e893933a4 P c613911f21464cdbbcab16b98107d740: Time spent starting tablet: real 0.006s user 0.000s sys 0.007s
I20260619 14:50:58.771538 20598 ts_tablet_manager.cc:1428] T 26d8dbba33a54ad49662054e893933a4 P 73d5892734b54ffab87341c35af43c74: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20260619 14:50:58.771870 20448 heartbeater.cc:503] Master 127.0.98.62:40379 was elected leader, sending a full tablet report...
I20260619 14:50:58.772509 20596 raft_consensus.cc:348] T 26d8dbba33a54ad49662054e893933a4 P 97e0feafcb89472bb177e25768077b1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c613911f21464cdbbcab16b98107d740" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43033 } } peers { permanent_uuid: "73d5892734b54ffab87341c35af43c74" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44161 } } peers { permanent_uuid: "97e0feafcb89472bb177e25768077b1b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41559 } }
I20260619 14:50:58.772830 20596 raft_consensus.cc:374] T 26d8dbba33a54ad49662054e893933a4 P 97e0feafcb89472bb177e25768077b1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:50:58.772898 20596 raft_consensus.cc:729] T 26d8dbba33a54ad49662054e893933a4 P 97e0feafcb89472bb177e25768077b1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97e0feafcb89472bb177e25768077b1b, State: Initialized, Role: FOLLOWER
I20260619 14:50:58.773604 20596 consensus_queue.cc:260] T 26d8dbba33a54ad49662054e893933a4 P 97e0feafcb89472bb177e25768077b1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c613911f21464cdbbcab16b98107d740" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43033 } } peers { permanent_uuid: "73d5892734b54ffab87341c35af43c74" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44161 } } peers { permanent_uuid: "97e0feafcb89472bb177e25768077b1b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41559 } }
I20260619 14:50:58.774369 20579 heartbeater.cc:503] Master 127.0.98.62:40379 was elected leader, sending a full tablet report...
I20260619 14:50:58.774669 20596 ts_tablet_manager.cc:1428] T 26d8dbba33a54ad49662054e893933a4 P 97e0feafcb89472bb177e25768077b1b: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
W20260619 14:50:58.942899 20580 tablet.cc:2369] T 26d8dbba33a54ad49662054e893933a4 P 97e0feafcb89472bb177e25768077b1b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 14:50:58.970942 20449 tablet.cc:2369] T 26d8dbba33a54ad49662054e893933a4 P 73d5892734b54ffab87341c35af43c74: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260619 14:50:59.009740 20318 tablet.cc:2369] T 26d8dbba33a54ad49662054e893933a4 P c613911f21464cdbbcab16b98107d740: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:50:59.101717 20603 raft_consensus.cc:482] T 26d8dbba33a54ad49662054e893933a4 P 73d5892734b54ffab87341c35af43c74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:50:59.101960 20603 raft_consensus.cc:504] T 26d8dbba33a54ad49662054e893933a4 P 73d5892734b54ffab87341c35af43c74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c613911f21464cdbbcab16b98107d740" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43033 } } peers { permanent_uuid: "73d5892734b54ffab87341c35af43c74" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44161 } } peers { permanent_uuid: "97e0feafcb89472bb177e25768077b1b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41559 } }
I20260619 14:50:59.103353 20603 leader_election.cc:288] T 26d8dbba33a54ad49662054e893933a4 P 73d5892734b54ffab87341c35af43c74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c613911f21464cdbbcab16b98107d740 (127.0.98.1:43033), 97e0feafcb89472bb177e25768077b1b (127.0.98.3:41559)
I20260619 14:50:59.109619 20595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:59.104132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59975 (local address 127.0.98.3:41559)
0619 14:50:59.104293 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:59.104296 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:59.104555 (+ 259us) server_negotiation.cc:407] Connection header received
0619 14:50:59.104839 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:59.104844 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:59.104916 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:59.105017 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:50:59.105285 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:59.105970 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:59.107634 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:59.108699 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:59.108803 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:59.108863 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:59.109399 (+ 536us) server_negotiation.cc:299] Negotiation successful
0619 14:50:59.109490 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260619 14:50:59.109606 20605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:59.103575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43033 (local address 127.0.98.2:58991)
0619 14:50:59.103862 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:59.103881 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:59.104101 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:59.104578 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:59.104582 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:59.104606 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:50:59.104813 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:59.104821 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:59.105864 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:59.105868 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:59.107437 (+ 1569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:59.107449 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:59.109022 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:59.109024 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:59.109187 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:59.109241 (+ 54us) client_negotiation.cc:769] Sending connection context
0619 14:50:59.109330 (+ 89us) client_negotiation.cc:240] Negotiation successful
0619 14:50:59.109415 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"mutex_wait_us":137,"spinlock_wait_cycles":4864,"thread_start_us":130,"threads_started":1}
I20260619 14:50:59.109794 20606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:59.103819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41559 (local address 127.0.98.2:59975)
0619 14:50:59.104370 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:59.104385 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:50:59.104700 (+ 315us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:50:59.105027 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:50:59.105030 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:50:59.105045 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0619 14:50:59.105165 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:59.105171 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:59.106081 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:59.106084 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:59.107437 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:50:59.107449 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:50:59.109011 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:50:59.109017 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:50:59.109193 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:59.109234 (+ 41us) client_negotiation.cc:769] Sending connection context
0619 14:50:59.109330 (+ 96us) client_negotiation.cc:240] Negotiation successful
0619 14:50:59.109565 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"mutex_wait_us":86,"spinlock_wait_cycles":2816,"thread_start_us":99,"threads_started":1}
I20260619 14:50:59.110246 20592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:59.103749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58991 (local address 127.0.98.1:43033)
0619 14:50:59.103996 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:59.104001 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:59.104023 (+ 22us) server_negotiation.cc:407] Connection header received
0619 14:50:59.104310 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:59.104315 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:59.104370 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:59.104487 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0619 14:50:59.104956 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:59.105750 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:59.107634 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:59.108511 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:59.108751 (+ 240us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:59.108832 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:59.109486 (+ 654us) server_negotiation.cc:299] Negotiation successful
0619 14:50:59.109573 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260619 14:50:59.110528 20533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26d8dbba33a54ad49662054e893933a4" candidate_uuid: "73d5892734b54ffab87341c35af43c74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97e0feafcb89472bb177e25768077b1b" is_pre_election: true
I20260619 14:50:59.110854 20533 raft_consensus.cc:2454] T 26d8dbba33a54ad49662054e893933a4 P 97e0feafcb89472bb177e25768077b1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73d5892734b54ffab87341c35af43c74 in term 0.
I20260619 14:50:59.110821 20271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26d8dbba33a54ad49662054e893933a4" candidate_uuid: "73d5892734b54ffab87341c35af43c74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c613911f21464cdbbcab16b98107d740" is_pre_election: true
I20260619 14:50:59.111169 20271 raft_consensus.cc:2454] T 26d8dbba33a54ad49662054e893933a4 P c613911f21464cdbbcab16b98107d740 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73d5892734b54ffab87341c35af43c74 in term 0.
I20260619 14:50:59.111485 20337 leader_election.cc:302] T 26d8dbba33a54ad49662054e893933a4 P 73d5892734b54ffab87341c35af43c74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73d5892734b54ffab87341c35af43c74, 97e0feafcb89472bb177e25768077b1b; no voters:
I20260619 14:50:59.111783 20603 raft_consensus.cc:2781] T 26d8dbba33a54ad49662054e893933a4 P 73d5892734b54ffab87341c35af43c74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260619 14:50:59.111876 20603 raft_consensus.cc:482] T 26d8dbba33a54ad49662054e893933a4 P 73d5892734b54ffab87341c35af43c74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260619 14:50:59.111908 20603 raft_consensus.cc:3037] T 26d8dbba33a54ad49662054e893933a4 P 73d5892734b54ffab87341c35af43c74 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:50:59.113459 20603 raft_consensus.cc:504] T 26d8dbba33a54ad49662054e893933a4 P 73d5892734b54ffab87341c35af43c74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c613911f21464cdbbcab16b98107d740" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43033 } } peers { permanent_uuid: "73d5892734b54ffab87341c35af43c74" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44161 } } peers { permanent_uuid: "97e0feafcb89472bb177e25768077b1b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41559 } }
I20260619 14:50:59.113920 20603 leader_election.cc:288] T 26d8dbba33a54ad49662054e893933a4 P 73d5892734b54ffab87341c35af43c74 [CANDIDATE]: Term 1 election: Requested vote from peers c613911f21464cdbbcab16b98107d740 (127.0.98.1:43033), 97e0feafcb89472bb177e25768077b1b (127.0.98.3:41559)
I20260619 14:50:59.114334 20271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26d8dbba33a54ad49662054e893933a4" candidate_uuid: "73d5892734b54ffab87341c35af43c74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c613911f21464cdbbcab16b98107d740"
I20260619 14:50:59.114411 20533 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26d8dbba33a54ad49662054e893933a4" candidate_uuid: "73d5892734b54ffab87341c35af43c74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97e0feafcb89472bb177e25768077b1b"
I20260619 14:50:59.114460 20271 raft_consensus.cc:3037] T 26d8dbba33a54ad49662054e893933a4 P c613911f21464cdbbcab16b98107d740 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:50:59.114571 20533 raft_consensus.cc:3037] T 26d8dbba33a54ad49662054e893933a4 P 97e0feafcb89472bb177e25768077b1b [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:50:59.115517 20271 raft_consensus.cc:2454] T 26d8dbba33a54ad49662054e893933a4 P c613911f21464cdbbcab16b98107d740 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73d5892734b54ffab87341c35af43c74 in term 1.
I20260619 14:50:59.115739 20533 raft_consensus.cc:2454] T 26d8dbba33a54ad49662054e893933a4 P 97e0feafcb89472bb177e25768077b1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73d5892734b54ffab87341c35af43c74 in term 1.
I20260619 14:50:59.115995 20336 leader_election.cc:302] T 26d8dbba33a54ad49662054e893933a4 P 73d5892734b54ffab87341c35af43c74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73d5892734b54ffab87341c35af43c74, c613911f21464cdbbcab16b98107d740; no voters:
I20260619 14:50:59.116195 20603 raft_consensus.cc:2781] T 26d8dbba33a54ad49662054e893933a4 P 73d5892734b54ffab87341c35af43c74 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:50:59.116473 20603 raft_consensus.cc:686] T 26d8dbba33a54ad49662054e893933a4 P 73d5892734b54ffab87341c35af43c74 [term 1 LEADER]: Becoming Leader. State: Replica: 73d5892734b54ffab87341c35af43c74, State: Running, Role: LEADER
I20260619 14:50:59.116901 20603 consensus_queue.cc:237] T 26d8dbba33a54ad49662054e893933a4 P 73d5892734b54ffab87341c35af43c74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c613911f21464cdbbcab16b98107d740" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43033 } } peers { permanent_uuid: "73d5892734b54ffab87341c35af43c74" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44161 } } peers { permanent_uuid: "97e0feafcb89472bb177e25768077b1b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41559 } }
I20260619 14:50:59.120494 20129 catalog_manager.cc:5495] T 26d8dbba33a54ad49662054e893933a4 P 73d5892734b54ffab87341c35af43c74 reported cstate change: term changed from 0 to 1, leader changed from <none> to 73d5892734b54ffab87341c35af43c74 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "73d5892734b54ffab87341c35af43c74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c613911f21464cdbbcab16b98107d740" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73d5892734b54ffab87341c35af43c74" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44161 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97e0feafcb89472bb177e25768077b1b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41559 } health_report { overall_health: UNKNOWN } } }
I20260619 14:50:59.204784 20594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:59.200794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48206 (local address 127.0.98.2:44161)
0619 14:50:59.200961 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0619 14:50:59.200968 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:50:59.200997 (+ 29us) server_negotiation.cc:407] Connection header received
0619 14:50:59.201103 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:50:59.201105 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:50:59.201168 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:50:59.201301 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:50:59.201670 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:59.202191 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:59.203102 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:50:59.203369 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:50:59.203507 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:50:59.203857 (+ 350us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:50:59.204315 (+ 458us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:50:59.204458 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0619 14:50:59.204531 (+ 73us) server_negotiation.cc:299] Negotiation successful
0619 14:50:59.204593 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260619 14:50:59.211874 20533 raft_consensus.cc:1261] T 26d8dbba33a54ad49662054e893933a4 P 97e0feafcb89472bb177e25768077b1b [term 1 FOLLOWER]: Refusing update from remote peer 73d5892734b54ffab87341c35af43c74: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:50:59.212001 20271 raft_consensus.cc:1261] T 26d8dbba33a54ad49662054e893933a4 P c613911f21464cdbbcab16b98107d740 [term 1 FOLLOWER]: Refusing update from remote peer 73d5892734b54ffab87341c35af43c74: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:50:59.213086 20603 consensus_queue.cc:1035] T 26d8dbba33a54ad49662054e893933a4 P 73d5892734b54ffab87341c35af43c74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c613911f21464cdbbcab16b98107d740" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:50:59.213357 20607 consensus_queue.cc:1035] T 26d8dbba33a54ad49662054e893933a4 P 73d5892734b54ffab87341c35af43c74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97e0feafcb89472bb177e25768077b1b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:50:59.222038 20611 mvcc.cc:205] Tried to move back new op lower bound from 7298583180122390528 to 7298583179746082816. Current Snapshot: MvccSnapshot[applied={T|T < 7298583180122390528}]
I20260619 14:50:59.224267 20612 mvcc.cc:205] Tried to move back new op lower bound from 7298583180122390528 to 7298583179746082816. Current Snapshot: MvccSnapshot[applied={T|T < 7298583180122390528}]
I20260619 14:50:59.233357 20127 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44916:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:50:59.233592 20127 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44916:
table { table_name: "test-table" } modify_external_catalogs: true
I20260619 14:50:59.236860 20127 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6be54523d86d478db9467c48383f7ecd: Sending DeleteTablet for 3 replicas of tablet 26d8dbba33a54ad49662054e893933a4
I20260619 14:50:59.238101 20382 tablet_service.cc:1514] Processing DeleteTablet for tablet 26d8dbba33a54ad49662054e893933a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:50:59 UTC) from {username='slave'} at 127.0.0.1:35342
I20260619 14:50:59.238044 20513 tablet_service.cc:1514] Processing DeleteTablet for tablet 26d8dbba33a54ad49662054e893933a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:50:59 UTC) from {username='slave'} at 127.0.0.1:37762
I20260619 14:50:59.238134 20251 tablet_service.cc:1514] Processing DeleteTablet for tablet 26d8dbba33a54ad49662054e893933a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-19 14:50:59 UTC) from {username='slave'} at 127.0.0.1:52430
I20260619 14:50:59.239233 20622 tablet_replica.cc:330] T 26d8dbba33a54ad49662054e893933a4 P c613911f21464cdbbcab16b98107d740: stopping tablet replica
I20260619 14:50:59.239709 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 20189
I20260619 14:50:59.239785 20622 raft_consensus.cc:2229] T 26d8dbba33a54ad49662054e893933a4 P c613911f21464cdbbcab16b98107d740 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:50:59.240404 20620 tablet_replica.cc:330] T 26d8dbba33a54ad49662054e893933a4 P 73d5892734b54ffab87341c35af43c74: stopping tablet replica
I20260619 14:50:59.240747 20620 raft_consensus.cc:2229] T 26d8dbba33a54ad49662054e893933a4 P 73d5892734b54ffab87341c35af43c74 [term 1 LEADER]: Raft consensus shutting down.
I20260619 14:50:59.240945 20621 tablet_replica.cc:330] T 26d8dbba33a54ad49662054e893933a4 P 97e0feafcb89472bb177e25768077b1b: stopping tablet replica
I20260619 14:50:59.241196 20620 raft_consensus.cc:2258] T 26d8dbba33a54ad49662054e893933a4 P 73d5892734b54ffab87341c35af43c74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:50:59.241333 20621 raft_consensus.cc:2229] T 26d8dbba33a54ad49662054e893933a4 P 97e0feafcb89472bb177e25768077b1b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260619 14:50:59.241822 20621 raft_consensus.cc:2258] T 26d8dbba33a54ad49662054e893933a4 P 97e0feafcb89472bb177e25768077b1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260619 14:50:59.243968 20621 ts_tablet_manager.cc:1905] T 26d8dbba33a54ad49662054e893933a4 P 97e0feafcb89472bb177e25768077b1b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:50:59.246479 20620 ts_tablet_manager.cc:1905] T 26d8dbba33a54ad49662054e893933a4 P 73d5892734b54ffab87341c35af43c74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260619 14:50:59.248818 20621 ts_tablet_manager.cc:1918] T 26d8dbba33a54ad49662054e893933a4 P 97e0feafcb89472bb177e25768077b1b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260619 14:50:59.248968 20621 log.cc:1196] T 26d8dbba33a54ad49662054e893933a4 P 97e0feafcb89472bb177e25768077b1b: Deleting WAL directory at /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781880383132139-392-0/minicluster-data/ts-2/wal/wals/26d8dbba33a54ad49662054e893933a4
I20260619 14:50:59.249364 20621 ts_tablet_manager.cc:1939] T 26d8dbba33a54ad49662054e893933a4 P 97e0feafcb89472bb177e25768077b1b: Deleting consensus metadata
W20260619 14:50:59.249902 20114 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43033 (error 108)
I20260619 14:50:59.250108 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 20320
I20260619 14:50:59.251895 20115 catalog_manager.cc:4841] TS 97e0feafcb89472bb177e25768077b1b (127.0.98.3:41559): tablet 26d8dbba33a54ad49662054e893933a4 (table test-table [id=8956e742fa2948299420f98e939c1e87]) successfully deleted
I20260619 14:50:59.252461 20624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:59.252021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43033 (local address 127.0.0.1:37434)
0619 14:50:59.252251 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:59.252343 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":63,"threads_started":1}
W20260619 14:50:59.252689 20114 catalog_manager.cc:4584] TS c613911f21464cdbbcab16b98107d740 (127.0.98.1:43033): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 26d8dbba33a54ad49662054e893933a4: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43033: connect: Connection refused (error 111)
I20260619 14:50:59.256881 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 20451
I20260619 14:50:59.257901 20624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:50:59.257121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44161 (local address 127.0.0.1:48220)
0619 14:50:59.257702 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0619 14:50:59.257772 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:44161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":473}
I20260619 14:50:59.263825 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 20098
2026-06-19T14:50:59Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1517 ms)
[----------] 6 tests from EncryptionPolicyTest (12051 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781880383132139-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:50:59 dist-test-slave-2sb1 krb5kdc[20627](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:50:59 dist-test-slave-2sb1 krb5kdc[20627](info): set up 2 sockets
Jun 19 14:50:59 dist-test-slave-2sb1 krb5kdc[20627](info): commencing operation
krb5kdc: starting...
W20260619 14:51:00.373471 392 mini_kdc.cc:121] Time spent starting KDC: real 1.086s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781880383132139-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781880383132139-392-0/krb5kdc/test-user.keytab.
Jun 19 14:51:00 dist-test-slave-2sb1 krb5kdc[20627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880660, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:51:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:51:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781880383132139-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781880383132139-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 14:51:00.517005 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781880383132139-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781880383132139-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781880383132139-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39535
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781880383132139-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39535
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:51:00.639038 20640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:51:00.639461 20640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:51:00.639552 20640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:51:00.644008 20640 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:51:00.644138 20640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:51:00.644178 20640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:51:00.644217 20640 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:51:00.644249 20640 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:51:00.651266 20640 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781880383132139-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781880383132139-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39535
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781880383132139-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39535
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781880383132139-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:51:00.652871 20640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:51:00.654124 20640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:51:00.664680 20648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:51:00.664590 20646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:51:00.664590 20645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:51:00.665727 20640 server_base.cc:1034] running on GCE node
I20260619 14:51:00.667006 20640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:51:00.668545 20640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:51:00.669875 20640 hybrid_clock.cc:648] HybridClock initialized: now 1781880660669811 us; error 89 us; skew 500 ppm
Jun 19 14:51:00 dist-test-slave-2sb1 krb5kdc[20627](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880660, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:51:00.673533 20640 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 14:51:00.674758 20640 webserver.cc:460] Webserver started at http://127.0.0.1:36785/ using document root <none> and password file <none>
I20260619 14:51:00.675414 20640 fs_manager.cc:359] Metadata directory not provided
I20260619 14:51:00.675514 20640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:51:00.675755 20640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:51:00.678052 20640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781880383132139-392-0/minicluster-data/master-0/data/instance:
uuid: "2256c95f48494f7ea4c2a712248f3365"
format_stamp: "Formatted at 2026-06-19 14:51:00 on dist-test-slave-2sb1"
server_key: "63f00bd6a1bf1b9d4c57ef6ad6cb3e49"
server_key_iv: "6641d3434e6b9412d6688594c9e42c0d"
server_key_version: "encryptionkey@0"
I20260619 14:51:00.678673 20640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781880383132139-392-0/minicluster-data/master-0/wal/instance:
uuid: "2256c95f48494f7ea4c2a712248f3365"
format_stamp: "Formatted at 2026-06-19 14:51:00 on dist-test-slave-2sb1"
server_key: "63f00bd6a1bf1b9d4c57ef6ad6cb3e49"
server_key_iv: "6641d3434e6b9412d6688594c9e42c0d"
server_key_version: "encryptionkey@0"
I20260619 14:51:00.682956 20640 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 14:51:00.685976 20655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:00.687451 20640 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:51:00.687659 20640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781880383132139-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781880383132139-392-0/minicluster-data/master-0/wal
uuid: "2256c95f48494f7ea4c2a712248f3365"
format_stamp: "Formatted at 2026-06-19 14:51:00 on dist-test-slave-2sb1"
server_key: "63f00bd6a1bf1b9d4c57ef6ad6cb3e49"
server_key_iv: "6641d3434e6b9412d6688594c9e42c0d"
server_key_version: "encryptionkey@0"
I20260619 14:51:00.687808 20640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781880383132139-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781880383132139-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781880383132139-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:51:00.706339 20640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:51:00.709961 20640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:51:00.710206 20640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:51:00.718361 20640 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39535
I20260619 14:51:00.718549 20707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39535 every 8 connection(s)
I20260619 14:51:00.719448 20640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:51:00.722662 20708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:51:00.725250 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 20640
I20260619 14:51:00.725446 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781880383132139-392-0/minicluster-data/master-0/wal/instance
I20260619 14:51:00.725742 392 external_mini_cluster.cc:1428] Setting key 49da21fc8b9531b7667dc540fce11463
I20260619 14:51:00.729779 20708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2256c95f48494f7ea4c2a712248f3365: Bootstrap starting.
I20260619 14:51:00.732532 20708 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2256c95f48494f7ea4c2a712248f3365: Neither blocks nor log segments found. Creating new log.
I20260619 14:51:00.733469 20708 log.cc:824] T 00000000000000000000000000000000 P 2256c95f48494f7ea4c2a712248f3365: Log is configured to *not* fsync() on all Append() calls
I20260619 14:51:00.735608 20708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2256c95f48494f7ea4c2a712248f3365: No bootstrap required, opened a new log
Jun 19 14:51:00 dist-test-slave-2sb1 krb5kdc[20627](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880660, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 14:51:00.740128 20708 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2256c95f48494f7ea4c2a712248f3365 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2256c95f48494f7ea4c2a712248f3365" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39535 } }
I20260619 14:51:00.740464 20708 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2256c95f48494f7ea4c2a712248f3365 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:51:00.740550 20708 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2256c95f48494f7ea4c2a712248f3365 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2256c95f48494f7ea4c2a712248f3365, State: Initialized, Role: FOLLOWER
I20260619 14:51:00.741169 20708 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2256c95f48494f7ea4c2a712248f3365 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2256c95f48494f7ea4c2a712248f3365" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39535 } }
I20260619 14:51:00.741377 20713 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2256c95f48494f7ea4c2a712248f3365 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260619 14:51:00.741536 20713 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2256c95f48494f7ea4c2a712248f3365 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2256c95f48494f7ea4c2a712248f3365" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39535 } }
I20260619 14:51:00.741356 20708 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2256c95f48494f7ea4c2a712248f3365 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:51:00.741953 20708 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2256c95f48494f7ea4c2a712248f3365 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:51:00.741948 20713 leader_election.cc:302] T 00000000000000000000000000000000 P 2256c95f48494f7ea4c2a712248f3365 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2256c95f48494f7ea4c2a712248f3365; no voters:
I20260619 14:51:00.742044 20708 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2256c95f48494f7ea4c2a712248f3365 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:51:00.742338 20713 leader_election.cc:288] T 00000000000000000000000000000000 P 2256c95f48494f7ea4c2a712248f3365 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers
I20260619 14:51:00.743319 20708 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2256c95f48494f7ea4c2a712248f3365 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2256c95f48494f7ea4c2a712248f3365" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39535 } }
I20260619 14:51:00.743484 20708 leader_election.cc:302] T 00000000000000000000000000000000 P 2256c95f48494f7ea4c2a712248f3365 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2256c95f48494f7ea4c2a712248f3365; no voters:
I20260619 14:51:00.743590 20708 leader_election.cc:288] T 00000000000000000000000000000000 P 2256c95f48494f7ea4c2a712248f3365 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:51:00.743609 20714 raft_consensus.cc:2741] T 00000000000000000000000000000000 P 2256c95f48494f7ea4c2a712248f3365 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260619 14:51:00.744025 20715 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2256c95f48494f7ea4c2a712248f3365 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:51:00.744176 20711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:00.727482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36405 (local address 127.0.0.1:39535)
0619 14:51:00.727942 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:00.727961 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:00.728005 (+ 44us) server_negotiation.cc:407] Connection header received
0619 14:51:00.728905 (+ 900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:00.728931 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:00.729369 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:00.729740 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:00.730619 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:00.732119 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:00.733232 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:00.733952 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:00.734077 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:00.736945 (+ 2868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:00.736953 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:00.736975 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:00.737008 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:00.739967 (+ 2959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:00.740797 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:00.740803 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:00.740808 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:00.740933 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:00.741308 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:00.741314 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:00.741320 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:00.742019 (+ 699us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:00.742247 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:00.742795 (+ 548us) server_negotiation.cc:299] Negotiation successful
0619 14:51:00.743050 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":122,"threads_started":1}
I20260619 14:51:00.744869 20708 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2256c95f48494f7ea4c2a712248f3365 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:51:00.745136 20715 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2256c95f48494f7ea4c2a712248f3365 [term 1 LEADER]: Becoming Leader. State: Replica: 2256c95f48494f7ea4c2a712248f3365, State: Running, Role: LEADER
I20260619 14:51:00.745484 20715 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2256c95f48494f7ea4c2a712248f3365 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2256c95f48494f7ea4c2a712248f3365" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39535 } }
I20260619 14:51:00.747016 20713 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2256c95f48494f7ea4c2a712248f3365 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2256c95f48494f7ea4c2a712248f3365" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2256c95f48494f7ea4c2a712248f3365" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39535 } } }
I20260619 14:51:00.747491 20713 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2256c95f48494f7ea4c2a712248f3365 [sys.catalog]: This master's current role is: LEADER
I20260619 14:51:00.747253 20714 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2256c95f48494f7ea4c2a712248f3365 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2256c95f48494f7ea4c2a712248f3365. Latest consensus state: current_term: 1 leader_uuid: "2256c95f48494f7ea4c2a712248f3365" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2256c95f48494f7ea4c2a712248f3365" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39535 } } }
I20260619 14:51:00.747725 20714 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2256c95f48494f7ea4c2a712248f3365 [sys.catalog]: This master's current role is: LEADER
I20260619 14:51:00.748111 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260619 14:51:00.750576 20728 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2256c95f48494f7ea4c2a712248f3365: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:51:00.750662 20728 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:51:00.750927 20729 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:51:00.751770 20729 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:51:00.758258 20729 catalog_manager.cc:1347] Generated new cluster ID: 00ee22774b054c7a82864971151a0da3
I20260619 14:51:00.758335 20729 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:51:00.761844 20711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:00.750706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43127 (local address 127.0.0.1:39535)
0619 14:51:00.750904 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:00.750908 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:00.751002 (+ 94us) server_negotiation.cc:407] Connection header received
0619 14:51:00.751127 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:00.751130 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:00.751178 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:00.751260 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:00.752192 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:00.753050 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:00.754328 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:00.754691 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:00.754807 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:00.756776 (+ 1969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:00.756780 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:00.756783 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:00.756811 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:00.759801 (+ 2990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:00.760461 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:00.760464 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:00.760466 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:00.760549 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:00.760927 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:00.760930 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:00.760932 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:00.761129 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:00.761279 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:00.761579 (+ 300us) server_negotiation.cc:299] Negotiation successful
0619 14:51:00.761678 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260619 14:51:00.767257 20729 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:51:00.768616 20729 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:51:00.776903 20729 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2256c95f48494f7ea4c2a712248f3365: Generated new TSK 0
I20260619 14:51:00.777808 20729 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:51:00.786283 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 20640
W20260619 14:51:00.792057 20625 connection.cc:535] client connection to 127.0.0.1:39535 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39535): Connection reset by peer (error 104)
2026-06-19T14:51:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1519 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781880383132139-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:51:00 dist-test-slave-2sb1 krb5kdc[20740](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:51:00 dist-test-slave-2sb1 krb5kdc[20740](info): set up 2 sockets
Jun 19 14:51:00 dist-test-slave-2sb1 krb5kdc[20740](info): commencing operation
krb5kdc: starting...
W20260619 14:51:01.890223 392 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781880383132139-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781880383132139-392-0/krb5kdc/test-user.keytab.
Jun 19 14:51:01 dist-test-slave-2sb1 krb5kdc[20740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880661, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:51:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:51:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781880383132139-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781880383132139-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 14:51:02.047010 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781880383132139-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781880383132139-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781880383132139-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45983
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781880383132139-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45983
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:51:02.174096 20753 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:51:02.174477 20753 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:51:02.174574 20753 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:51:02.179123 20753 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:51:02.179253 20753 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:51:02.179292 20753 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:51:02.179332 20753 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:51:02.179363 20753 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:51:02.186090 20753 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781880383132139-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781880383132139-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45983
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781880383132139-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45983
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20753
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781880383132139-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:51:02.187359 20753 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:51:02.188508 20753 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 14:51:02.198664 20753 server_base.cc:1034] running on GCE node
W20260619 14:51:02.198691 20761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:51:02.198865 20758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:51:02.198626 20759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:51:02.199530 20753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:51:02.200872 20753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:51:02.202092 20753 hybrid_clock.cc:648] HybridClock initialized: now 1781880662202058 us; error 53 us; skew 500 ppm
Jun 19 14:51:02 dist-test-slave-2sb1 krb5kdc[20740](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880662, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:51:02.206101 20753 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 14:51:02.208488 20753 webserver.cc:460] Webserver started at http://127.0.0.1:35207/ using document root <none> and password file <none>
I20260619 14:51:02.209297 20753 fs_manager.cc:359] Metadata directory not provided
I20260619 14:51:02.209470 20753 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:51:02.209748 20753 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:51:02.212298 20753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781880383132139-392-0/minicluster-data/master-0/data/instance:
uuid: "6c68e963646541e28c5ab67de2aaebd8"
format_stamp: "Formatted at 2026-06-19 14:51:02 on dist-test-slave-2sb1"
server_key: "6e33225117fc52544f997e9f756b19d0"
server_key_iv: "5750ae7c3b65a882ff8d8f3be47c2b6c"
server_key_version: "encryptionkey@0"
I20260619 14:51:02.213030 20753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781880383132139-392-0/minicluster-data/master-0/wal/instance:
uuid: "6c68e963646541e28c5ab67de2aaebd8"
format_stamp: "Formatted at 2026-06-19 14:51:02 on dist-test-slave-2sb1"
server_key: "6e33225117fc52544f997e9f756b19d0"
server_key_iv: "5750ae7c3b65a882ff8d8f3be47c2b6c"
server_key_version: "encryptionkey@0"
I20260619 14:51:02.218264 20753 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 14:51:02.221509 20768 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:02.223227 20753 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:51:02.223589 20753 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781880383132139-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781880383132139-392-0/minicluster-data/master-0/wal
uuid: "6c68e963646541e28c5ab67de2aaebd8"
format_stamp: "Formatted at 2026-06-19 14:51:02 on dist-test-slave-2sb1"
server_key: "6e33225117fc52544f997e9f756b19d0"
server_key_iv: "5750ae7c3b65a882ff8d8f3be47c2b6c"
server_key_version: "encryptionkey@0"
I20260619 14:51:02.223752 20753 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781880383132139-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781880383132139-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781880383132139-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:51:02.243108 20753 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:51:02.247203 20753 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:51:02.247476 20753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:51:02.255566 20753 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45983
I20260619 14:51:02.255606 20820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45983 every 8 connection(s)
I20260619 14:51:02.256779 20753 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:51:02.260140 20821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:51:02.266656 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 20753
I20260619 14:51:02.266876 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781880383132139-392-0/minicluster-data/master-0/wal/instance
I20260619 14:51:02.267204 392 external_mini_cluster.cc:1428] Setting key 4419087b3dd6787e65b354b55f4133fa
I20260619 14:51:02.267693 20821 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c68e963646541e28c5ab67de2aaebd8: Bootstrap starting.
I20260619 14:51:02.270730 20821 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c68e963646541e28c5ab67de2aaebd8: Neither blocks nor log segments found. Creating new log.
I20260619 14:51:02.271709 20821 log.cc:824] T 00000000000000000000000000000000 P 6c68e963646541e28c5ab67de2aaebd8: Log is configured to *not* fsync() on all Append() calls
I20260619 14:51:02.274538 20821 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c68e963646541e28c5ab67de2aaebd8: No bootstrap required, opened a new log
I20260619 14:51:02.279227 20824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:02.269066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50331 (local address 127.0.0.1:45983)
0619 14:51:02.269537 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:02.269557 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:02.269601 (+ 44us) server_negotiation.cc:407] Connection header received
0619 14:51:02.270525 (+ 924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:02.270553 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:02.271044 (+ 491us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:02.271452 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:02.271904 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:02.273211 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:02.274352 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:02.275190 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:02.275388 (+ 198us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:02.276467 (+ 1079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:02.276474 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:02.276495 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:51:02.277158 (+ 663us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:02.277244 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:51:02.277277 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:02.277436 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:02.277666 (+ 230us) server_negotiation.cc:299] Negotiation successful
0619 14:51:02.277835 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":135,"threads_started":1}
I20260619 14:51:02.280133 20821 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c68e963646541e28c5ab67de2aaebd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c68e963646541e28c5ab67de2aaebd8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45983 } }
I20260619 14:51:02.280395 20821 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c68e963646541e28c5ab67de2aaebd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:51:02.280452 20821 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c68e963646541e28c5ab67de2aaebd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c68e963646541e28c5ab67de2aaebd8, State: Initialized, Role: FOLLOWER
I20260619 14:51:02.281046 20821 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c68e963646541e28c5ab67de2aaebd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c68e963646541e28c5ab67de2aaebd8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45983 } }
I20260619 14:51:02.281227 20821 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c68e963646541e28c5ab67de2aaebd8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:51:02.281301 20821 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c68e963646541e28c5ab67de2aaebd8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:51:02.281407 20821 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c68e963646541e28c5ab67de2aaebd8 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:51:02.282702 20821 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c68e963646541e28c5ab67de2aaebd8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c68e963646541e28c5ab67de2aaebd8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45983 } }
I20260619 14:51:02.283134 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:51:02.283129 20821 leader_election.cc:302] T 00000000000000000000000000000000 P 6c68e963646541e28c5ab67de2aaebd8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c68e963646541e28c5ab67de2aaebd8; no voters:
I20260619 14:51:02.283519 20821 leader_election.cc:288] T 00000000000000000000000000000000 P 6c68e963646541e28c5ab67de2aaebd8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:51:02.283677 20826 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c68e963646541e28c5ab67de2aaebd8 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:51:02.284034 20826 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c68e963646541e28c5ab67de2aaebd8 [term 1 LEADER]: Becoming Leader. State: Replica: 6c68e963646541e28c5ab67de2aaebd8, State: Running, Role: LEADER
I20260619 14:51:02.284403 20826 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c68e963646541e28c5ab67de2aaebd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c68e963646541e28c5ab67de2aaebd8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45983 } }
I20260619 14:51:02.285071 20821 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c68e963646541e28c5ab67de2aaebd8 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:51:02.285856 20826 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c68e963646541e28c5ab67de2aaebd8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6c68e963646541e28c5ab67de2aaebd8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c68e963646541e28c5ab67de2aaebd8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45983 } } }
I20260619 14:51:02.286015 20826 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c68e963646541e28c5ab67de2aaebd8 [sys.catalog]: This master's current role is: LEADER
I20260619 14:51:02.286216 20826 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c68e963646541e28c5ab67de2aaebd8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c68e963646541e28c5ab67de2aaebd8. Latest consensus state: current_term: 1 leader_uuid: "6c68e963646541e28c5ab67de2aaebd8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c68e963646541e28c5ab67de2aaebd8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45983 } } }
I20260619 14:51:02.286314 20826 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c68e963646541e28c5ab67de2aaebd8 [sys.catalog]: This master's current role is: LEADER
I20260619 14:51:02.286581 20839 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:51:02.290302 20839 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:51:02.294574 20824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:02.288261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58099 (local address 127.0.0.1:45983)
0619 14:51:02.288477 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:02.288482 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:02.289436 (+ 954us) server_negotiation.cc:407] Connection header received
0619 14:51:02.289639 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:02.289642 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:02.289696 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:02.289795 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:02.290128 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:02.290985 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:02.292494 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:02.292858 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:02.292925 (+ 67us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:02.294045 (+ 1120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:02.294048 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:02.294050 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:51:02.294065 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:02.294092 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:51:02.294104 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:02.294208 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:02.294370 (+ 162us) server_negotiation.cc:299] Negotiation successful
0619 14:51:02.294411 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260619 14:51:02.297099 20839 catalog_manager.cc:1347] Generated new cluster ID: 8652b8ad86c244ffa9976b53518f0d55
I20260619 14:51:02.297194 20839 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:51:02.317518 20839 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:51:02.318682 20839 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:51:02.325771 20839 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6c68e963646541e28c5ab67de2aaebd8: Generated new TSK 0
I20260619 14:51:02.326735 20839 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:51:02.340372 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 20753
W20260619 14:51:02.345966 20738 connection.cc:535] client connection to 127.0.0.1:45983 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45983): Connection reset by peer (error 104)
2026-06-19T14:51:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1553 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781880383132139-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:51:02 dist-test-slave-2sb1 krb5kdc[20852](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:51:02 dist-test-slave-2sb1 krb5kdc[20852](info): set up 2 sockets
Jun 19 14:51:02 dist-test-slave-2sb1 krb5kdc[20852](info): commencing operation
krb5kdc: starting...
W20260619 14:51:03.453552 392 mini_kdc.cc:121] Time spent starting KDC: real 1.093s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781880383132139-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781880383132139-392-0/krb5kdc/test-user.keytab.
Jun 19 14:51:03 dist-test-slave-2sb1 krb5kdc[20852](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880663, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:51:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:51:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781880383132139-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781880383132139-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 14:51:03.593338 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781880383132139-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781880383132139-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781880383132139-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35187
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781880383132139-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35187
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:51:03.718902 20865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:51:03.719285 20865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:51:03.719365 20865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:51:03.723793 20865 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:51:03.723930 20865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:51:03.723969 20865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:51:03.724010 20865 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:51:03.724046 20865 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:51:03.729858 20865 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781880383132139-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781880383132139-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35187
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781880383132139-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35187
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20865
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781880383132139-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:51:03.731076 20865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:51:03.732187 20865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:51:03.742242 20873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:51:03.742251 20870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:51:03.742300 20871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:51:03.742317 20865 server_base.cc:1034] running on GCE node
I20260619 14:51:03.743170 20865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:51:03.744472 20865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:51:03.745702 20865 hybrid_clock.cc:648] HybridClock initialized: now 1781880663745669 us; error 56 us; skew 500 ppm
Jun 19 14:51:03 dist-test-slave-2sb1 krb5kdc[20852](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880663, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:51:03.758203 20865 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 14:51:03.759250 20865 webserver.cc:460] Webserver started at http://127.0.0.1:38071/ using document root <none> and password file <none>
I20260619 14:51:03.759895 20865 fs_manager.cc:359] Metadata directory not provided
I20260619 14:51:03.759971 20865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:51:03.760156 20865 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:51:03.762533 20865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781880383132139-392-0/minicluster-data/master-0/data/instance:
uuid: "3ceb5b19fa1e404e87f72012801941af"
format_stamp: "Formatted at 2026-06-19 14:51:03 on dist-test-slave-2sb1"
server_key: "54a9b65a8187a8f95e6ae5d8551aaeac"
server_key_iv: "fa7409dba8d55b2cae0ba11f88f434bc"
server_key_version: "encryptionkey@0"
I20260619 14:51:03.763077 20865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781880383132139-392-0/minicluster-data/master-0/wal/instance:
uuid: "3ceb5b19fa1e404e87f72012801941af"
format_stamp: "Formatted at 2026-06-19 14:51:03 on dist-test-slave-2sb1"
server_key: "54a9b65a8187a8f95e6ae5d8551aaeac"
server_key_iv: "fa7409dba8d55b2cae0ba11f88f434bc"
server_key_version: "encryptionkey@0"
I20260619 14:51:03.767901 20865 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 14:51:03.771946 20880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:03.774031 20865 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260619 14:51:03.774187 20865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781880383132139-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781880383132139-392-0/minicluster-data/master-0/wal
uuid: "3ceb5b19fa1e404e87f72012801941af"
format_stamp: "Formatted at 2026-06-19 14:51:03 on dist-test-slave-2sb1"
server_key: "54a9b65a8187a8f95e6ae5d8551aaeac"
server_key_iv: "fa7409dba8d55b2cae0ba11f88f434bc"
server_key_version: "encryptionkey@0"
I20260619 14:51:03.774315 20865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781880383132139-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781880383132139-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781880383132139-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:51:03.779717 20865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:51:03.783959 20865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:51:03.784144 20865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:51:03.792156 20865 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35187
I20260619 14:51:03.792179 20932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35187 every 8 connection(s)
I20260619 14:51:03.793233 20865 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:51:03.796415 20933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:51:03.801784 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 20865
I20260619 14:51:03.801932 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781880383132139-392-0/minicluster-data/master-0/wal/instance
I20260619 14:51:03.802265 392 external_mini_cluster.cc:1428] Setting key 7e839c70abad82d37440cff27f308486
I20260619 14:51:03.804093 20933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ceb5b19fa1e404e87f72012801941af: Bootstrap starting.
I20260619 14:51:03.808308 20933 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3ceb5b19fa1e404e87f72012801941af: Neither blocks nor log segments found. Creating new log.
I20260619 14:51:03.809952 20933 log.cc:824] T 00000000000000000000000000000000 P 3ceb5b19fa1e404e87f72012801941af: Log is configured to *not* fsync() on all Append() calls
I20260619 14:51:03.810676 20936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:03.804026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42753 (local address 127.0.0.1:35187)
0619 14:51:03.804738 (+ 712us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:03.804757 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:03.804802 (+ 45us) server_negotiation.cc:407] Connection header received
0619 14:51:03.805791 (+ 989us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:03.805822 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:03.806389 (+ 567us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:03.806869 (+ 480us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:03.808352 (+ 1483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:03.808366 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:03.808401 (+ 35us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:51:03.808433 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:03.808572 (+ 139us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:51:03.808622 (+ 50us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:03.809035 (+ 413us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:03.809329 (+ 294us) server_negotiation.cc:299] Negotiation successful
0619 14:51:03.809538 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":518,"thread_start_us":182,"threads_started":1}
I20260619 14:51:03.813133 20933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3ceb5b19fa1e404e87f72012801941af: No bootstrap required, opened a new log
I20260619 14:51:03.814327 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:51:03.817646 20933 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3ceb5b19fa1e404e87f72012801941af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ceb5b19fa1e404e87f72012801941af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35187 } }
I20260619 14:51:03.818058 20933 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3ceb5b19fa1e404e87f72012801941af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:51:03.818130 20933 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3ceb5b19fa1e404e87f72012801941af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ceb5b19fa1e404e87f72012801941af, State: Initialized, Role: FOLLOWER
I20260619 14:51:03.818620 20933 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3ceb5b19fa1e404e87f72012801941af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ceb5b19fa1e404e87f72012801941af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35187 } }
I20260619 14:51:03.818791 20933 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3ceb5b19fa1e404e87f72012801941af [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:51:03.818886 20933 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3ceb5b19fa1e404e87f72012801941af [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:51:03.818996 20933 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3ceb5b19fa1e404e87f72012801941af [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:51:03.819797 20936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:03.817534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33707 (local address 127.0.0.1:35187)
0619 14:51:03.817690 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:03.817694 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:03.817733 (+ 39us) server_negotiation.cc:407] Connection header received
0619 14:51:03.817860 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:03.817863 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:03.817905 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:03.817991 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:03.819316 (+ 1325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:03.819320 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:03.819323 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:51:03.819335 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:03.819367 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:51:03.819384 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:03.819479 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:03.819623 (+ 144us) server_negotiation.cc:299] Negotiation successful
0619 14:51:03.819666 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 14:51:03.820425 20933 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3ceb5b19fa1e404e87f72012801941af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ceb5b19fa1e404e87f72012801941af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35187 } }
I20260619 14:51:03.820883 20933 leader_election.cc:302] T 00000000000000000000000000000000 P 3ceb5b19fa1e404e87f72012801941af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3ceb5b19fa1e404e87f72012801941af; no voters:
I20260619 14:51:03.821231 20933 leader_election.cc:288] T 00000000000000000000000000000000 P 3ceb5b19fa1e404e87f72012801941af [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:51:03.821828 20943 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3ceb5b19fa1e404e87f72012801941af [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:51:03.822089 20943 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3ceb5b19fa1e404e87f72012801941af [term 1 LEADER]: Becoming Leader. State: Replica: 3ceb5b19fa1e404e87f72012801941af, State: Running, Role: LEADER
I20260619 14:51:03.822481 20943 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3ceb5b19fa1e404e87f72012801941af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ceb5b19fa1e404e87f72012801941af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35187 } }
I20260619 14:51:03.823086 20933 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3ceb5b19fa1e404e87f72012801941af [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:51:03.824638 20944 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ceb5b19fa1e404e87f72012801941af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3ceb5b19fa1e404e87f72012801941af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ceb5b19fa1e404e87f72012801941af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35187 } } }
I20260619 14:51:03.824733 20945 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3ceb5b19fa1e404e87f72012801941af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3ceb5b19fa1e404e87f72012801941af. Latest consensus state: current_term: 1 leader_uuid: "3ceb5b19fa1e404e87f72012801941af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ceb5b19fa1e404e87f72012801941af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35187 } } }
I20260619 14:51:03.824851 20944 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ceb5b19fa1e404e87f72012801941af [sys.catalog]: This master's current role is: LEADER
I20260619 14:51:03.824934 20945 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3ceb5b19fa1e404e87f72012801941af [sys.catalog]: This master's current role is: LEADER
I20260619 14:51:03.825734 20952 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:51:03.829034 20952 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:51:03.834847 20952 catalog_manager.cc:1347] Generated new cluster ID: b03ad0b8b7ca405abd4d68ceb7f2968c
I20260619 14:51:03.834949 20952 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:51:03.853129 20952 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:51:03.853545 20952 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:51:03.858843 20952 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3ceb5b19fa1e404e87f72012801941af: Generated new TSK 0
I20260619 14:51:03.859784 20952 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:51:03.867635 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 20865
2026-06-19T14:51:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1527 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781880383132139-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:51:03 dist-test-slave-2sb1 krb5kdc[20964](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:51:03 dist-test-slave-2sb1 krb5kdc[20964](info): set up 2 sockets
Jun 19 14:51:03 dist-test-slave-2sb1 krb5kdc[20964](info): commencing operation
krb5kdc: starting...
W20260619 14:51:04.998103 392 mini_kdc.cc:121] Time spent starting KDC: real 1.110s user 0.007s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781880383132139-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781880383132139-392-0/krb5kdc/test-user.keytab.
Jun 19 14:51:05 dist-test-slave-2sb1 krb5kdc[20964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880665, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:51:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:51:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781880383132139-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781880383132139-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 14:51:05.149726 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781880383132139-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781880383132139-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781880383132139-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37715
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781880383132139-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37715
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:51:05.273739 20977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:51:05.274127 20977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:51:05.274211 20977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:51:05.279062 20977 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:51:05.279181 20977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:51:05.279218 20977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:51:05.279259 20977 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:51:05.279295 20977 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:51:05.285140 20977 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781880383132139-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781880383132139-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37715
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781880383132139-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37715
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20977
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781880383132139-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:51:05.286343 20977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:51:05.287497 20977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:51:05.297813 20983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:51:05.297801 20982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:51:05.297772 20985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:51:05.298103 20977 server_base.cc:1034] running on GCE node
I20260619 14:51:05.298928 20977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:51:05.300225 20977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:51:05.301477 20977 hybrid_clock.cc:648] HybridClock initialized: now 1781880665301417 us; error 80 us; skew 500 ppm
Jun 19 14:51:05 dist-test-slave-2sb1 krb5kdc[20964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880665, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:51:05.304934 20977 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 14:51:05.305961 20977 webserver.cc:460] Webserver started at http://127.0.0.1:37011/ using document root <none> and password file <none>
I20260619 14:51:05.306502 20977 fs_manager.cc:359] Metadata directory not provided
I20260619 14:51:05.306566 20977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:51:05.306748 20977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:51:05.308980 20977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781880383132139-392-0/minicluster-data/master-0/data/instance:
uuid: "b9154b76d6f142a9a76465f14a0759c6"
format_stamp: "Formatted at 2026-06-19 14:51:05 on dist-test-slave-2sb1"
server_key: "209d8e8c2dd75e92353ecf22a01ef499"
server_key_iv: "ff410abc749d3c66f6babd22e167e47c"
server_key_version: "encryptionkey@0"
I20260619 14:51:05.309548 20977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781880383132139-392-0/minicluster-data/master-0/wal/instance:
uuid: "b9154b76d6f142a9a76465f14a0759c6"
format_stamp: "Formatted at 2026-06-19 14:51:05 on dist-test-slave-2sb1"
server_key: "209d8e8c2dd75e92353ecf22a01ef499"
server_key_iv: "ff410abc749d3c66f6babd22e167e47c"
server_key_version: "encryptionkey@0"
I20260619 14:51:05.313792 20977 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 14:51:05.317648 20992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:05.319492 20977 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20260619 14:51:05.319774 20977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781880383132139-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781880383132139-392-0/minicluster-data/master-0/wal
uuid: "b9154b76d6f142a9a76465f14a0759c6"
format_stamp: "Formatted at 2026-06-19 14:51:05 on dist-test-slave-2sb1"
server_key: "209d8e8c2dd75e92353ecf22a01ef499"
server_key_iv: "ff410abc749d3c66f6babd22e167e47c"
server_key_version: "encryptionkey@0"
I20260619 14:51:05.319934 20977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781880383132139-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781880383132139-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781880383132139-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:51:05.336704 20977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:51:05.340706 20977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:51:05.340942 20977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:51:05.349531 20977 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37715
I20260619 14:51:05.349869 21044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37715 every 8 connection(s)
I20260619 14:51:05.350613 20977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:51:05.353783 21045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:51:05.357532 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 20977
I20260619 14:51:05.357715 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781880383132139-392-0/minicluster-data/master-0/wal/instance
I20260619 14:51:05.358013 392 external_mini_cluster.cc:1428] Setting key 0ab7a4a607fd74b81f14e5088a34deb3
I20260619 14:51:05.360507 21045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9154b76d6f142a9a76465f14a0759c6: Bootstrap starting.
I20260619 14:51:05.363343 21045 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b9154b76d6f142a9a76465f14a0759c6: Neither blocks nor log segments found. Creating new log.
I20260619 14:51:05.364259 21045 log.cc:824] T 00000000000000000000000000000000 P b9154b76d6f142a9a76465f14a0759c6: Log is configured to *not* fsync() on all Append() calls
I20260619 14:51:05.367590 21045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9154b76d6f142a9a76465f14a0759c6: No bootstrap required, opened a new log
Jun 19 14:51:05 dist-test-slave-2sb1 krb5kdc[20964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880665, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 14:51:05.371964 21045 raft_consensus.cc:348] T 00000000000000000000000000000000 P b9154b76d6f142a9a76465f14a0759c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9154b76d6f142a9a76465f14a0759c6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37715 } }
I20260619 14:51:05.372267 21045 raft_consensus.cc:374] T 00000000000000000000000000000000 P b9154b76d6f142a9a76465f14a0759c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:51:05.372330 21045 raft_consensus.cc:729] T 00000000000000000000000000000000 P b9154b76d6f142a9a76465f14a0759c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9154b76d6f142a9a76465f14a0759c6, State: Initialized, Role: FOLLOWER
I20260619 14:51:05.372967 21045 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9154b76d6f142a9a76465f14a0759c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9154b76d6f142a9a76465f14a0759c6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37715 } }
I20260619 14:51:05.373142 21045 raft_consensus.cc:388] T 00000000000000000000000000000000 P b9154b76d6f142a9a76465f14a0759c6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:51:05.373211 21045 raft_consensus.cc:482] T 00000000000000000000000000000000 P b9154b76d6f142a9a76465f14a0759c6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:51:05.373310 21045 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b9154b76d6f142a9a76465f14a0759c6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:51:05.374678 21045 raft_consensus.cc:504] T 00000000000000000000000000000000 P b9154b76d6f142a9a76465f14a0759c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9154b76d6f142a9a76465f14a0759c6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37715 } }
I20260619 14:51:05.375178 21045 leader_election.cc:302] T 00000000000000000000000000000000 P b9154b76d6f142a9a76465f14a0759c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b9154b76d6f142a9a76465f14a0759c6; no voters:
I20260619 14:51:05.375535 21048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:05.359567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54535 (local address 127.0.0.1:37715)
0619 14:51:05.360005 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:05.360020 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:05.360055 (+ 35us) server_negotiation.cc:407] Connection header received
0619 14:51:05.360947 (+ 892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:05.360969 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:05.361360 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:05.361689 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:05.362668 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:05.363757 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:05.365293 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:05.365999 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:05.366139 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:05.369769 (+ 3630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:05.369775 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:05.369790 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:05.369824 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:05.371777 (+ 1953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:05.372312 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:05.372337 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:05.372345 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:05.372517 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:05.372856 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:05.372859 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:05.372861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:05.373486 (+ 625us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:05.373700 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:05.374035 (+ 335us) server_negotiation.cc:299] Negotiation successful
0619 14:51:05.374365 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":146,"threads_started":1}
I20260619 14:51:05.375566 21045 leader_election.cc:288] T 00000000000000000000000000000000 P b9154b76d6f142a9a76465f14a0759c6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:51:05.375739 21050 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b9154b76d6f142a9a76465f14a0759c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:51:05.376012 21050 raft_consensus.cc:686] T 00000000000000000000000000000000 P b9154b76d6f142a9a76465f14a0759c6 [term 1 LEADER]: Becoming Leader. State: Replica: b9154b76d6f142a9a76465f14a0759c6, State: Running, Role: LEADER
I20260619 14:51:05.376396 21050 consensus_queue.cc:237] T 00000000000000000000000000000000 P b9154b76d6f142a9a76465f14a0759c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9154b76d6f142a9a76465f14a0759c6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37715 } }
I20260619 14:51:05.376902 21045 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9154b76d6f142a9a76465f14a0759c6 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:51:05.377770 21050 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9154b76d6f142a9a76465f14a0759c6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9154b76d6f142a9a76465f14a0759c6. Latest consensus state: current_term: 1 leader_uuid: "b9154b76d6f142a9a76465f14a0759c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9154b76d6f142a9a76465f14a0759c6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37715 } } }
I20260619 14:51:05.378433 21052 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9154b76d6f142a9a76465f14a0759c6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b9154b76d6f142a9a76465f14a0759c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9154b76d6f142a9a76465f14a0759c6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37715 } } }
I20260619 14:51:05.378569 21050 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9154b76d6f142a9a76465f14a0759c6 [sys.catalog]: This master's current role is: LEADER
I20260619 14:51:05.378708 21052 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9154b76d6f142a9a76465f14a0759c6 [sys.catalog]: This master's current role is: LEADER
I20260619 14:51:05.379175 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260619 14:51:05.382513 21065 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b9154b76d6f142a9a76465f14a0759c6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:51:05.382601 21065 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:51:05.382692 21062 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:51:05.383661 21062 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:51:05.390663 21062 catalog_manager.cc:1347] Generated new cluster ID: 91b0bb0817844f909cbde81636f8ff1a
I20260619 14:51:05.390750 21062 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:51:05.393121 21048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:05.383246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60457 (local address 127.0.0.1:37715)
0619 14:51:05.383479 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:05.383483 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:05.383725 (+ 242us) server_negotiation.cc:407] Connection header received
0619 14:51:05.383937 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:05.383940 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:05.383985 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:05.384029 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:05.384944 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:05.385492 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:05.386669 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:05.387074 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:05.387203 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:05.389143 (+ 1940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:05.389147 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:05.389150 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:05.389183 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:05.391331 (+ 2148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:05.391808 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:05.391811 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:05.391813 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:05.391895 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:05.392173 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:05.392177 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:05.392179 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:05.392363 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:05.392503 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:05.392825 (+ 322us) server_negotiation.cc:299] Negotiation successful
0619 14:51:05.392954 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260619 14:51:05.402051 21062 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:51:05.403185 21062 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:51:05.413335 21062 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b9154b76d6f142a9a76465f14a0759c6: Generated new TSK 0
I20260619 14:51:05.414139 21062 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:51:05.421542 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 20977
2026-06-19T14:51:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1553 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781880383132139-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:51:05 dist-test-slave-2sb1 krb5kdc[21076](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:51:05 dist-test-slave-2sb1 krb5kdc[21076](info): set up 2 sockets
Jun 19 14:51:05 dist-test-slave-2sb1 krb5kdc[21076](info): commencing operation
krb5kdc: starting...
W20260619 14:51:06.553659 392 mini_kdc.cc:121] Time spent starting KDC: real 1.112s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781880383132139-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781880383132139-392-0/krb5kdc/test-user.keytab.
Jun 19 14:51:06 dist-test-slave-2sb1 krb5kdc[21076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:51:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:51:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781880383132139-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781880383132139-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 14:51:06.723472 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781880383132139-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781880383132139-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781880383132139-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36711
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781880383132139-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36711
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:51:06.847143 21089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:51:06.847558 21089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:51:06.847653 21089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:51:06.851897 21089 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:51:06.852023 21089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:51:06.852073 21089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:51:06.852133 21089 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:51:06.852190 21089 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:51:06.858098 21089 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781880383132139-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781880383132139-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36711
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781880383132139-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21089
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781880383132139-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:51:06.859969 21089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:51:06.861236 21089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:51:06.870858 21097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:51:06.871030 21094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:51:06.871030 21089 server_base.cc:1034] running on GCE node
W20260619 14:51:06.871197 21095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:51:06.872035 21089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:51:06.873402 21089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:51:06.874636 21089 hybrid_clock.cc:648] HybridClock initialized: now 1781880666874560 us; error 94 us; skew 500 ppm
Jun 19 14:51:06 dist-test-slave-2sb1 krb5kdc[21076](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880666, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:51:06.877971 21089 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 14:51:06.879172 21089 webserver.cc:460] Webserver started at http://127.0.0.1:34909/ using document root <none> and password file <none>
I20260619 14:51:06.880141 21089 fs_manager.cc:359] Metadata directory not provided
I20260619 14:51:06.880267 21089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:51:06.880578 21089 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:51:06.882923 21089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781880383132139-392-0/minicluster-data/master-0/data/instance:
uuid: "3aada496ffaf4ae798631ac6cc611f80"
format_stamp: "Formatted at 2026-06-19 14:51:06 on dist-test-slave-2sb1"
server_key: "4801d9fb079312cd994aabef7d77000f"
server_key_iv: "9e78049dcd7f9c8c1e85e6067fe2f19a"
server_key_version: "encryptionkey@0"
I20260619 14:51:06.883569 21089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781880383132139-392-0/minicluster-data/master-0/wal/instance:
uuid: "3aada496ffaf4ae798631ac6cc611f80"
format_stamp: "Formatted at 2026-06-19 14:51:06 on dist-test-slave-2sb1"
server_key: "4801d9fb079312cd994aabef7d77000f"
server_key_iv: "9e78049dcd7f9c8c1e85e6067fe2f19a"
server_key_version: "encryptionkey@0"
I20260619 14:51:06.887863 21089 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260619 14:51:06.890841 21104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:06.892516 21089 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260619 14:51:06.892757 21089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781880383132139-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781880383132139-392-0/minicluster-data/master-0/wal
uuid: "3aada496ffaf4ae798631ac6cc611f80"
format_stamp: "Formatted at 2026-06-19 14:51:06 on dist-test-slave-2sb1"
server_key: "4801d9fb079312cd994aabef7d77000f"
server_key_iv: "9e78049dcd7f9c8c1e85e6067fe2f19a"
server_key_version: "encryptionkey@0"
I20260619 14:51:06.892910 21089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781880383132139-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781880383132139-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781880383132139-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:51:06.914090 21089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:51:06.917829 21089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:51:06.918068 21089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:51:06.926479 21089 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36711
I20260619 14:51:06.926585 21156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36711 every 8 connection(s)
I20260619 14:51:06.927597 21089 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:51:06.931432 21157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:51:06.932276 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 21089
I20260619 14:51:06.932422 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781880383132139-392-0/minicluster-data/master-0/wal/instance
I20260619 14:51:06.932767 392 external_mini_cluster.cc:1428] Setting key 622bf3d12db938e7b36081c5575d2a25
I20260619 14:51:06.940203 21157 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3aada496ffaf4ae798631ac6cc611f80: Bootstrap starting.
I20260619 14:51:06.942922 21157 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3aada496ffaf4ae798631ac6cc611f80: Neither blocks nor log segments found. Creating new log.
I20260619 14:51:06.943754 21157 log.cc:824] T 00000000000000000000000000000000 P 3aada496ffaf4ae798631ac6cc611f80: Log is configured to *not* fsync() on all Append() calls
I20260619 14:51:06.945411 21160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:06.934994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53373 (local address 127.0.0.1:36711)
0619 14:51:06.935555 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:06.935583 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:06.935622 (+ 39us) server_negotiation.cc:407] Connection header received
0619 14:51:06.936280 (+ 658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:06.936299 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:06.936742 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:06.937150 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:06.938534 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:06.940304 (+ 1770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:06.941587 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:06.942204 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:06.942406 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:06.943412 (+ 1006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:06.943420 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:06.943441 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:51:06.943940 (+ 499us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:06.944006 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:51:06.944032 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:06.944144 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:06.944268 (+ 124us) server_negotiation.cc:299] Negotiation successful
0619 14:51:06.944510 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":209,"threads_started":1}
I20260619 14:51:06.946007 21157 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3aada496ffaf4ae798631ac6cc611f80: No bootstrap required, opened a new log
I20260619 14:51:06.948881 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:51:06.950939 21157 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3aada496ffaf4ae798631ac6cc611f80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3aada496ffaf4ae798631ac6cc611f80" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36711 } }
I20260619 14:51:06.951311 21157 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3aada496ffaf4ae798631ac6cc611f80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:51:06.951378 21157 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3aada496ffaf4ae798631ac6cc611f80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aada496ffaf4ae798631ac6cc611f80, State: Initialized, Role: FOLLOWER
I20260619 14:51:06.951877 21157 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3aada496ffaf4ae798631ac6cc611f80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3aada496ffaf4ae798631ac6cc611f80" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36711 } }
I20260619 14:51:06.952040 21157 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3aada496ffaf4ae798631ac6cc611f80 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:51:06.952149 21157 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3aada496ffaf4ae798631ac6cc611f80 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:51:06.952270 21157 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3aada496ffaf4ae798631ac6cc611f80 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:51:06.953534 21157 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3aada496ffaf4ae798631ac6cc611f80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3aada496ffaf4ae798631ac6cc611f80" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36711 } }
I20260619 14:51:06.953962 21157 leader_election.cc:302] T 00000000000000000000000000000000 P 3aada496ffaf4ae798631ac6cc611f80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3aada496ffaf4ae798631ac6cc611f80; no voters:
I20260619 14:51:06.954346 21157 leader_election.cc:288] T 00000000000000000000000000000000 P 3aada496ffaf4ae798631ac6cc611f80 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:51:06.954480 21167 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3aada496ffaf4ae798631ac6cc611f80 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:51:06.954774 21167 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3aada496ffaf4ae798631ac6cc611f80 [term 1 LEADER]: Becoming Leader. State: Replica: 3aada496ffaf4ae798631ac6cc611f80, State: Running, Role: LEADER
I20260619 14:51:06.955189 21167 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3aada496ffaf4ae798631ac6cc611f80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3aada496ffaf4ae798631ac6cc611f80" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36711 } }
I20260619 14:51:06.955917 21157 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3aada496ffaf4ae798631ac6cc611f80 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:51:06.957100 21160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:06.951577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51223 (local address 127.0.0.1:36711)
0619 14:51:06.951705 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:06.951712 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:06.951780 (+ 68us) server_negotiation.cc:407] Connection header received
0619 14:51:06.951941 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:06.951946 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:06.951995 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:06.952095 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:06.952489 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:06.953101 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:06.954436 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:06.954847 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:06.954964 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:06.956166 (+ 1202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:06.956170 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:06.956173 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:51:06.956188 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:06.956219 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:51:06.956232 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:06.956315 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:06.956839 (+ 524us) server_negotiation.cc:299] Negotiation successful
0619 14:51:06.956928 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 14:51:06.957038 21169 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3aada496ffaf4ae798631ac6cc611f80 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3aada496ffaf4ae798631ac6cc611f80. Latest consensus state: current_term: 1 leader_uuid: "3aada496ffaf4ae798631ac6cc611f80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3aada496ffaf4ae798631ac6cc611f80" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36711 } } }
I20260619 14:51:06.957167 21169 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3aada496ffaf4ae798631ac6cc611f80 [sys.catalog]: This master's current role is: LEADER
I20260619 14:51:06.957507 21168 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3aada496ffaf4ae798631ac6cc611f80 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3aada496ffaf4ae798631ac6cc611f80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3aada496ffaf4ae798631ac6cc611f80" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36711 } } }
I20260619 14:51:06.957599 21168 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3aada496ffaf4ae798631ac6cc611f80 [sys.catalog]: This master's current role is: LEADER
W20260619 14:51:06.961555 21183 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3aada496ffaf4ae798631ac6cc611f80: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:51:06.961664 21183 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:51:06.961745 21176 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:51:06.962493 21176 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:51:06.968030 21176 catalog_manager.cc:1347] Generated new cluster ID: 51fda264b6e44c4883d3ec4069a5ed34
I20260619 14:51:06.968127 21176 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:51:06.981966 21176 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:51:06.983036 21176 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:51:06.987906 21176 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3aada496ffaf4ae798631ac6cc611f80: Generated new TSK 0
I20260619 14:51:06.988821 21176 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:51:07.001292 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 21089
2026-06-19T14:51:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1580 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781880383132139-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:51:07 dist-test-slave-2sb1 krb5kdc[21188](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:51:07 dist-test-slave-2sb1 krb5kdc[21188](info): set up 2 sockets
Jun 19 14:51:07 dist-test-slave-2sb1 krb5kdc[21188](info): commencing operation
krb5kdc: starting...
W20260619 14:51:08.101442 392 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781880383132139-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781880383132139-392-0/krb5kdc/test-user.keytab.
Jun 19 14:51:08 dist-test-slave-2sb1 krb5kdc[21188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880668, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:51:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:51:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781880383132139-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781880383132139-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 14:51:08.249984 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781880383132139-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781880383132139-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781880383132139-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41609
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781880383132139-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41609
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:51:08.379809 21201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:51:08.380113 21201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:51:08.380158 21201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:51:08.384416 21201 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:51:08.384536 21201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:51:08.384572 21201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:51:08.384591 21201 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:51:08.384608 21201 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:51:08.390648 21201 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781880383132139-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781880383132139-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41609
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781880383132139-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41609
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21201
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781880383132139-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:51:08.391831 21201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:51:08.392875 21201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:51:08.402381 21206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:51:08.402549 21201 server_base.cc:1034] running on GCE node
W20260619 14:51:08.402484 21209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:51:08.402732 21207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:51:08.403477 21201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:51:08.404978 21201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:51:08.406219 21201 hybrid_clock.cc:648] HybridClock initialized: now 1781880668406177 us; error 54 us; skew 500 ppm
Jun 19 14:51:08 dist-test-slave-2sb1 krb5kdc[21188](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880668, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:51:08.410283 21201 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 14:51:08.411784 21201 webserver.cc:460] Webserver started at http://127.0.0.1:41009/ using document root <none> and password file <none>
I20260619 14:51:08.412513 21201 fs_manager.cc:359] Metadata directory not provided
I20260619 14:51:08.412623 21201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:51:08.412909 21201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:51:08.415174 21201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781880383132139-392-0/minicluster-data/master-0/data/instance:
uuid: "07a6e55f3d6f4eafaa537f06f6dc933a"
format_stamp: "Formatted at 2026-06-19 14:51:08 on dist-test-slave-2sb1"
server_key: "0dfb7ed20999a09b2da6f1b613f1bda6"
server_key_iv: "33c963cdc5c483201a79108719a0be0a"
server_key_version: "encryptionkey@0"
I20260619 14:51:08.415831 21201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781880383132139-392-0/minicluster-data/master-0/wal/instance:
uuid: "07a6e55f3d6f4eafaa537f06f6dc933a"
format_stamp: "Formatted at 2026-06-19 14:51:08 on dist-test-slave-2sb1"
server_key: "0dfb7ed20999a09b2da6f1b613f1bda6"
server_key_iv: "33c963cdc5c483201a79108719a0be0a"
server_key_version: "encryptionkey@0"
I20260619 14:51:08.420464 21201 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 14:51:08.423434 21216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:08.425053 21201 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:51:08.425261 21201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781880383132139-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781880383132139-392-0/minicluster-data/master-0/wal
uuid: "07a6e55f3d6f4eafaa537f06f6dc933a"
format_stamp: "Formatted at 2026-06-19 14:51:08 on dist-test-slave-2sb1"
server_key: "0dfb7ed20999a09b2da6f1b613f1bda6"
server_key_iv: "33c963cdc5c483201a79108719a0be0a"
server_key_version: "encryptionkey@0"
I20260619 14:51:08.425407 21201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781880383132139-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781880383132139-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781880383132139-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:51:08.430997 21201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:51:08.435185 21201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:51:08.435441 21201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:51:08.443643 21201 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41609
I20260619 14:51:08.443719 21268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41609 every 8 connection(s)
I20260619 14:51:08.444801 21201 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:51:08.447893 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 21201
I20260619 14:51:08.448094 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781880383132139-392-0/minicluster-data/master-0/wal/instance
I20260619 14:51:08.448086 21269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:51:08.448385 392 external_mini_cluster.cc:1428] Setting key 27d154f823b38ab1078cdb9c39db978c
I20260619 14:51:08.455117 21272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:08.450023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53671 (local address 127.0.0.1:41609)
0619 14:51:08.450528 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:08.450548 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:08.450592 (+ 44us) server_negotiation.cc:407] Connection header received
0619 14:51:08.451796 (+ 1204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:08.451822 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:08.452221 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:08.452610 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:08.453459 (+ 849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:08.453465 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:08.453482 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:51:08.453495 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:08.453571 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:51:08.453599 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:08.453758 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:08.453980 (+ 222us) server_negotiation.cc:299] Negotiation successful
0619 14:51:08.454140 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":169,"threads_started":1}
I20260619 14:51:08.455921 21269 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07a6e55f3d6f4eafaa537f06f6dc933a: Bootstrap starting.
I20260619 14:51:08.458626 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:51:08.458674 21269 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 07a6e55f3d6f4eafaa537f06f6dc933a: Neither blocks nor log segments found. Creating new log.
I20260619 14:51:08.459559 21269 log.cc:824] T 00000000000000000000000000000000 P 07a6e55f3d6f4eafaa537f06f6dc933a: Log is configured to *not* fsync() on all Append() calls
I20260619 14:51:08.462105 21269 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07a6e55f3d6f4eafaa537f06f6dc933a: No bootstrap required, opened a new log
I20260619 14:51:08.463447 21272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:08.461085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54997 (local address 127.0.0.1:41609)
0619 14:51:08.461232 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:08.461240 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:08.461564 (+ 324us) server_negotiation.cc:407] Connection header received
0619 14:51:08.461683 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:08.461685 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:08.461795 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:08.461873 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:08.462831 (+ 958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:08.462834 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:08.462836 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:51:08.462847 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:08.462873 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:51:08.462884 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:08.462965 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:08.463113 (+ 148us) server_negotiation.cc:299] Negotiation successful
0619 14:51:08.463283 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 14:51:08.466449 21269 raft_consensus.cc:348] T 00000000000000000000000000000000 P 07a6e55f3d6f4eafaa537f06f6dc933a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07a6e55f3d6f4eafaa537f06f6dc933a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41609 } }
I20260619 14:51:08.466713 21269 raft_consensus.cc:374] T 00000000000000000000000000000000 P 07a6e55f3d6f4eafaa537f06f6dc933a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:51:08.466756 21269 raft_consensus.cc:729] T 00000000000000000000000000000000 P 07a6e55f3d6f4eafaa537f06f6dc933a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07a6e55f3d6f4eafaa537f06f6dc933a, State: Initialized, Role: FOLLOWER
I20260619 14:51:08.467252 21269 consensus_queue.cc:260] T 00000000000000000000000000000000 P 07a6e55f3d6f4eafaa537f06f6dc933a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07a6e55f3d6f4eafaa537f06f6dc933a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41609 } }
I20260619 14:51:08.467381 21269 raft_consensus.cc:388] T 00000000000000000000000000000000 P 07a6e55f3d6f4eafaa537f06f6dc933a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:51:08.467471 21269 raft_consensus.cc:482] T 00000000000000000000000000000000 P 07a6e55f3d6f4eafaa537f06f6dc933a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:51:08.467630 21269 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 07a6e55f3d6f4eafaa537f06f6dc933a [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:51:08.468772 21269 raft_consensus.cc:504] T 00000000000000000000000000000000 P 07a6e55f3d6f4eafaa537f06f6dc933a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07a6e55f3d6f4eafaa537f06f6dc933a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41609 } }
I20260619 14:51:08.469163 21269 leader_election.cc:302] T 00000000000000000000000000000000 P 07a6e55f3d6f4eafaa537f06f6dc933a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 07a6e55f3d6f4eafaa537f06f6dc933a; no voters:
I20260619 14:51:08.469542 21269 leader_election.cc:288] T 00000000000000000000000000000000 P 07a6e55f3d6f4eafaa537f06f6dc933a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:51:08.469638 21279 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 07a6e55f3d6f4eafaa537f06f6dc933a [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:51:08.469822 21279 raft_consensus.cc:686] T 00000000000000000000000000000000 P 07a6e55f3d6f4eafaa537f06f6dc933a [term 1 LEADER]: Becoming Leader. State: Replica: 07a6e55f3d6f4eafaa537f06f6dc933a, State: Running, Role: LEADER
I20260619 14:51:08.470196 21279 consensus_queue.cc:237] T 00000000000000000000000000000000 P 07a6e55f3d6f4eafaa537f06f6dc933a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07a6e55f3d6f4eafaa537f06f6dc933a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41609 } }
I20260619 14:51:08.470966 21269 sys_catalog.cc:564] T 00000000000000000000000000000000 P 07a6e55f3d6f4eafaa537f06f6dc933a [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:51:08.472208 21280 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07a6e55f3d6f4eafaa537f06f6dc933a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "07a6e55f3d6f4eafaa537f06f6dc933a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07a6e55f3d6f4eafaa537f06f6dc933a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41609 } } }
I20260619 14:51:08.472352 21280 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07a6e55f3d6f4eafaa537f06f6dc933a [sys.catalog]: This master's current role is: LEADER
I20260619 14:51:08.472707 21281 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07a6e55f3d6f4eafaa537f06f6dc933a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07a6e55f3d6f4eafaa537f06f6dc933a. Latest consensus state: current_term: 1 leader_uuid: "07a6e55f3d6f4eafaa537f06f6dc933a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07a6e55f3d6f4eafaa537f06f6dc933a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41609 } } }
I20260619 14:51:08.472805 21288 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:51:08.472874 21281 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07a6e55f3d6f4eafaa537f06f6dc933a [sys.catalog]: This master's current role is: LEADER
I20260619 14:51:08.476502 21288 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:51:08.482455 21288 catalog_manager.cc:1347] Generated new cluster ID: 5155847591da49faac8a2d91556afd8e
I20260619 14:51:08.482550 21288 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:51:08.497836 21288 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:51:08.498238 21288 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:51:08.503587 21288 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 07a6e55f3d6f4eafaa537f06f6dc933a: Generated new TSK 0
I20260619 14:51:08.504467 21288 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:51:08.515921 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 21201
2026-06-19T14:51:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1513 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781880383132139-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:51:08 dist-test-slave-2sb1 krb5kdc[21300](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:51:08 dist-test-slave-2sb1 krb5kdc[21300](info): set up 2 sockets
Jun 19 14:51:08 dist-test-slave-2sb1 krb5kdc[21300](info): commencing operation
krb5kdc: starting...
W20260619 14:51:09.653729 392 mini_kdc.cc:121] Time spent starting KDC: real 1.117s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781880383132139-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781880383132139-392-0/krb5kdc/test-user.keytab.
Jun 19 14:51:09 dist-test-slave-2sb1 krb5kdc[21300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880669, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:51:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:51:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781880383132139-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781880383132139-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 14:51:09.797411 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781880383132139-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781880383132139-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781880383132139-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32919
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781880383132139-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32919
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:51:09.922163 21313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:51:09.922487 21313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:51:09.922533 21313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:51:09.927145 21313 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:51:09.927238 21313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:51:09.927259 21313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:51:09.927277 21313 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:51:09.927294 21313 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:51:09.933377 21313 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781880383132139-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781880383132139-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32919
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781880383132139-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32919
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21313
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781880383132139-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:51:09.934588 21313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:51:09.935662 21313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:51:09.946009 21321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:51:09.946439 21319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:51:09.946388 21318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:51:09.946918 21313 server_base.cc:1034] running on GCE node
I20260619 14:51:09.947598 21313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:51:09.949148 21313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:51:09.950498 21313 hybrid_clock.cc:648] HybridClock initialized: now 1781880669950357 us; error 159 us; skew 500 ppm
Jun 19 14:51:09 dist-test-slave-2sb1 krb5kdc[21300](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880669, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:51:09.958904 21313 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 14:51:09.960052 21313 webserver.cc:460] Webserver started at http://127.0.0.1:35199/ using document root <none> and password file <none>
I20260619 14:51:09.961002 21313 fs_manager.cc:359] Metadata directory not provided
I20260619 14:51:09.961126 21313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:51:09.961372 21313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:51:09.963639 21313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781880383132139-392-0/minicluster-data/master-0/data/instance:
uuid: "c1bbe97b1de2408498a87882336b995b"
format_stamp: "Formatted at 2026-06-19 14:51:09 on dist-test-slave-2sb1"
server_key: "bcb178c0d49fce2cebe0459587f846da"
server_key_iv: "92ffe7ab1ef8b99ef82fc9995eccd4b1"
server_key_version: "encryptionkey@0"
I20260619 14:51:09.964309 21313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781880383132139-392-0/minicluster-data/master-0/wal/instance:
uuid: "c1bbe97b1de2408498a87882336b995b"
format_stamp: "Formatted at 2026-06-19 14:51:09 on dist-test-slave-2sb1"
server_key: "bcb178c0d49fce2cebe0459587f846da"
server_key_iv: "92ffe7ab1ef8b99ef82fc9995eccd4b1"
server_key_version: "encryptionkey@0"
I20260619 14:51:09.969050 21313 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260619 14:51:09.972265 21328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:09.974148 21313 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260619 14:51:09.974476 21313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781880383132139-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781880383132139-392-0/minicluster-data/master-0/wal
uuid: "c1bbe97b1de2408498a87882336b995b"
format_stamp: "Formatted at 2026-06-19 14:51:09 on dist-test-slave-2sb1"
server_key: "bcb178c0d49fce2cebe0459587f846da"
server_key_iv: "92ffe7ab1ef8b99ef82fc9995eccd4b1"
server_key_version: "encryptionkey@0"
I20260619 14:51:09.974632 21313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781880383132139-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781880383132139-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781880383132139-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:51:09.990767 21313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:51:09.994270 21313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:51:09.994509 21313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:51:10.002154 21313 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32919
I20260619 14:51:10.002180 21380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32919 every 8 connection(s)
I20260619 14:51:10.003201 21313 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:51:10.005618 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 21313
I20260619 14:51:10.005793 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781880383132139-392-0/minicluster-data/master-0/wal/instance
I20260619 14:51:10.006115 392 external_mini_cluster.cc:1428] Setting key 969b52eafeb5e406c1ca6fbfadd26cf0
I20260619 14:51:10.006664 21381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:51:10.015003 21381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1bbe97b1de2408498a87882336b995b: Bootstrap starting.
Jun 19 14:51:10 dist-test-slave-2sb1 krb5kdc[21300](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880669, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 14:51:10.017843 21381 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c1bbe97b1de2408498a87882336b995b: Neither blocks nor log segments found. Creating new log.
I20260619 14:51:10.018814 21381 log.cc:824] T 00000000000000000000000000000000 P c1bbe97b1de2408498a87882336b995b: Log is configured to *not* fsync() on all Append() calls
I20260619 14:51:10.021162 21381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1bbe97b1de2408498a87882336b995b: No bootstrap required, opened a new log
I20260619 14:51:10.023362 21384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:10.008451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.164:47901 (local address 127.0.0.1:32919)
0619 14:51:10.008976 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:10.008993 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:10.009031 (+ 38us) server_negotiation.cc:407] Connection header received
0619 14:51:10.010192 (+ 1161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:10.010216 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:10.010658 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:10.011063 (+ 405us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:10.011984 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:10.013082 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:10.014007 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:10.014517 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:10.014701 (+ 184us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:10.017795 (+ 3094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:10.017800 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:10.017815 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:10.017850 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:10.020070 (+ 2220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:10.020763 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:10.020769 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:10.020775 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:10.020893 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:10.021299 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:10.021303 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:10.021304 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:10.021756 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:10.021989 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:10.022351 (+ 362us) server_negotiation.cc:299] Negotiation successful
0619 14:51:10.022581 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":235,"threads_started":1}
I20260619 14:51:10.026196 21381 raft_consensus.cc:348] T 00000000000000000000000000000000 P c1bbe97b1de2408498a87882336b995b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1bbe97b1de2408498a87882336b995b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32919 } }
I20260619 14:51:10.026571 21381 raft_consensus.cc:374] T 00000000000000000000000000000000 P c1bbe97b1de2408498a87882336b995b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:51:10.026635 21381 raft_consensus.cc:729] T 00000000000000000000000000000000 P c1bbe97b1de2408498a87882336b995b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1bbe97b1de2408498a87882336b995b, State: Initialized, Role: FOLLOWER
I20260619 14:51:10.027190 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:51:10.027206 21381 consensus_queue.cc:260] T 00000000000000000000000000000000 P c1bbe97b1de2408498a87882336b995b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1bbe97b1de2408498a87882336b995b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32919 } }
I20260619 14:51:10.027443 21381 raft_consensus.cc:388] T 00000000000000000000000000000000 P c1bbe97b1de2408498a87882336b995b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:51:10.027541 21381 raft_consensus.cc:482] T 00000000000000000000000000000000 P c1bbe97b1de2408498a87882336b995b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:51:10.027655 21381 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c1bbe97b1de2408498a87882336b995b [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:51:10.028927 21381 raft_consensus.cc:504] T 00000000000000000000000000000000 P c1bbe97b1de2408498a87882336b995b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1bbe97b1de2408498a87882336b995b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32919 } }
I20260619 14:51:10.029341 21381 leader_election.cc:302] T 00000000000000000000000000000000 P c1bbe97b1de2408498a87882336b995b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c1bbe97b1de2408498a87882336b995b; no voters:
I20260619 14:51:10.029731 21381 leader_election.cc:288] T 00000000000000000000000000000000 P c1bbe97b1de2408498a87882336b995b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:51:10.029832 21390 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c1bbe97b1de2408498a87882336b995b [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:51:10.030148 21390 raft_consensus.cc:686] T 00000000000000000000000000000000 P c1bbe97b1de2408498a87882336b995b [term 1 LEADER]: Becoming Leader. State: Replica: c1bbe97b1de2408498a87882336b995b, State: Running, Role: LEADER
I20260619 14:51:10.030515 21390 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1bbe97b1de2408498a87882336b995b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1bbe97b1de2408498a87882336b995b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32919 } }
I20260619 14:51:10.031421 21381 sys_catalog.cc:564] T 00000000000000000000000000000000 P c1bbe97b1de2408498a87882336b995b [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:51:10.032145 21392 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1bbe97b1de2408498a87882336b995b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c1bbe97b1de2408498a87882336b995b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1bbe97b1de2408498a87882336b995b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32919 } } }
I20260619 14:51:10.032279 21392 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1bbe97b1de2408498a87882336b995b [sys.catalog]: This master's current role is: LEADER
I20260619 14:51:10.032713 21393 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1bbe97b1de2408498a87882336b995b [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1bbe97b1de2408498a87882336b995b. Latest consensus state: current_term: 1 leader_uuid: "c1bbe97b1de2408498a87882336b995b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1bbe97b1de2408498a87882336b995b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32919 } } }
I20260619 14:51:10.032888 21393 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1bbe97b1de2408498a87882336b995b [sys.catalog]: This master's current role is: LEADER
I20260619 14:51:10.034324 21397 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:51:10.038277 21397 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:51:10.040367 21384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:10.029610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.164:43737 (local address 127.0.0.1:32919)
0619 14:51:10.029764 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:10.029770 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:10.030328 (+ 558us) server_negotiation.cc:407] Connection header received
0619 14:51:10.030378 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:10.030381 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:10.030433 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:10.030505 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:10.031544 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:10.032418 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:10.033822 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:10.034112 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:10.034233 (+ 121us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:10.035912 (+ 1679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:10.035916 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:10.035919 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:10.035950 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:10.037970 (+ 2020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:10.038620 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:10.038624 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:10.038626 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:10.038738 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:10.039298 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:10.039302 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:10.039303 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:10.039544 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:10.039717 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:10.040079 (+ 362us) server_negotiation.cc:299] Negotiation successful
0619 14:51:10.040196 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260619 14:51:10.045853 21397 catalog_manager.cc:1347] Generated new cluster ID: b6003a5082064e67b521bbf7ac0cd886
I20260619 14:51:10.045979 21397 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:51:10.076776 21397 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:51:10.078331 21397 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:51:10.093026 21397 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c1bbe97b1de2408498a87882336b995b: Generated new TSK 0
I20260619 14:51:10.094067 21397 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:51:10.102988 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 21313
W20260619 14:51:10.109131 21298 connection.cc:535] client connection to 127.0.0.1:32919 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:32919): Connection reset by peer (error 104)
2026-06-19T14:51:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1587 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781880383132139-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:51:10 dist-test-slave-2sb1 krb5kdc[21412](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:51:10 dist-test-slave-2sb1 krb5kdc[21412](info): set up 2 sockets
Jun 19 14:51:10 dist-test-slave-2sb1 krb5kdc[21412](info): commencing operation
krb5kdc: starting...
W20260619 14:51:11.221468 392 mini_kdc.cc:121] Time spent starting KDC: real 1.097s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781880383132139-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781880383132139-392-0/krb5kdc/test-user.keytab.
Jun 19 14:51:11 dist-test-slave-2sb1 krb5kdc[21412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880671, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:51:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:51:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781880383132139-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781880383132139-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 14:51:11.380169 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781880383132139-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781880383132139-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781880383132139-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39831
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781880383132139-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39831
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:51:11.519170 21425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:51:11.519526 21425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:51:11.519603 21425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:51:11.524377 21425 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:51:11.524528 21425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:51:11.524569 21425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:51:11.524607 21425 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:51:11.524669 21425 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:51:11.530840 21425 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781880383132139-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781880383132139-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39831
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781880383132139-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21425
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781880383132139-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:51:11.532044 21425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:51:11.533162 21425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:51:11.544006 21430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:51:11.544032 21433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:51:11.544268 21431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:51:11.544413 21425 server_base.cc:1034] running on GCE node
I20260619 14:51:11.545202 21425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:51:11.546900 21425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:51:11.548166 21425 hybrid_clock.cc:648] HybridClock initialized: now 1781880671548130 us; error 66 us; skew 500 ppm
Jun 19 14:51:11 dist-test-slave-2sb1 krb5kdc[21412](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880671, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:51:11.552053 21425 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 14:51:11.553299 21425 webserver.cc:460] Webserver started at http://127.0.0.1:42621/ using document root <none> and password file <none>
I20260619 14:51:11.553987 21425 fs_manager.cc:359] Metadata directory not provided
I20260619 14:51:11.554087 21425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:51:11.554337 21425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:51:11.556579 21425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781880383132139-392-0/minicluster-data/master-0/data/instance:
uuid: "f31a89976b964b35a016585642a66a0f"
format_stamp: "Formatted at 2026-06-19 14:51:11 on dist-test-slave-2sb1"
server_key: "28a038ffe4f3af05e65a13f9a4803cc3"
server_key_iv: "d17743dd7c82f0e0aa0db5b7e7494f84"
server_key_version: "encryptionkey@0"
I20260619 14:51:11.557284 21425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781880383132139-392-0/minicluster-data/master-0/wal/instance:
uuid: "f31a89976b964b35a016585642a66a0f"
format_stamp: "Formatted at 2026-06-19 14:51:11 on dist-test-slave-2sb1"
server_key: "28a038ffe4f3af05e65a13f9a4803cc3"
server_key_iv: "d17743dd7c82f0e0aa0db5b7e7494f84"
server_key_version: "encryptionkey@0"
I20260619 14:51:11.561533 21425 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 14:51:11.564606 21440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:11.566504 21425 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:51:11.566718 21425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781880383132139-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781880383132139-392-0/minicluster-data/master-0/wal
uuid: "f31a89976b964b35a016585642a66a0f"
format_stamp: "Formatted at 2026-06-19 14:51:11 on dist-test-slave-2sb1"
server_key: "28a038ffe4f3af05e65a13f9a4803cc3"
server_key_iv: "d17743dd7c82f0e0aa0db5b7e7494f84"
server_key_version: "encryptionkey@0"
I20260619 14:51:11.566871 21425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781880383132139-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781880383132139-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781880383132139-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:51:11.579124 21425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:51:11.582981 21425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:51:11.583230 21425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:51:11.592329 21425 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39831
I20260619 14:51:11.592367 21492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39831 every 8 connection(s)
I20260619 14:51:11.593470 21425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:51:11.596628 21493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:51:11.600860 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 21425
I20260619 14:51:11.601043 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781880383132139-392-0/minicluster-data/master-0/wal/instance
I20260619 14:51:11.601414 392 external_mini_cluster.cc:1428] Setting key 028a12d5ced9852fcc7039d38eaa16e9
I20260619 14:51:11.603791 21493 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f31a89976b964b35a016585642a66a0f: Bootstrap starting.
I20260619 14:51:11.607317 21493 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f31a89976b964b35a016585642a66a0f: Neither blocks nor log segments found. Creating new log.
I20260619 14:51:11.608348 21493 log.cc:824] T 00000000000000000000000000000000 P f31a89976b964b35a016585642a66a0f: Log is configured to *not* fsync() on all Append() calls
I20260619 14:51:11.610924 21493 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f31a89976b964b35a016585642a66a0f: No bootstrap required, opened a new log
I20260619 14:51:11.614187 21496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:11.603494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.164:49083 (local address 127.0.0.1:39831)
0619 14:51:11.604046 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:11.604066 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:11.604114 (+ 48us) server_negotiation.cc:407] Connection header received
0619 14:51:11.605076 (+ 962us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:11.605108 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:11.605603 (+ 495us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:11.606206 (+ 603us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:11.606801 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:11.608471 (+ 1670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:11.609720 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:11.610323 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:11.610460 (+ 137us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:11.611695 (+ 1235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:11.611700 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:11.611716 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:51:11.612170 (+ 454us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:11.612235 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:51:11.612260 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:11.612383 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:11.613158 (+ 775us) server_negotiation.cc:299] Negotiation successful
0619 14:51:11.613337 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":181,"threads_started":1}
I20260619 14:51:11.616030 21493 raft_consensus.cc:348] T 00000000000000000000000000000000 P f31a89976b964b35a016585642a66a0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f31a89976b964b35a016585642a66a0f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39831 } }
I20260619 14:51:11.616448 21493 raft_consensus.cc:374] T 00000000000000000000000000000000 P f31a89976b964b35a016585642a66a0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:51:11.616564 21493 raft_consensus.cc:729] T 00000000000000000000000000000000 P f31a89976b964b35a016585642a66a0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f31a89976b964b35a016585642a66a0f, State: Initialized, Role: FOLLOWER
I20260619 14:51:11.617077 21493 consensus_queue.cc:260] T 00000000000000000000000000000000 P f31a89976b964b35a016585642a66a0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f31a89976b964b35a016585642a66a0f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39831 } }
I20260619 14:51:11.617216 21493 raft_consensus.cc:388] T 00000000000000000000000000000000 P f31a89976b964b35a016585642a66a0f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:51:11.617328 21493 raft_consensus.cc:482] T 00000000000000000000000000000000 P f31a89976b964b35a016585642a66a0f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:51:11.617445 21493 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f31a89976b964b35a016585642a66a0f [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:51:11.617795 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:51:11.618806 21493 raft_consensus.cc:504] T 00000000000000000000000000000000 P f31a89976b964b35a016585642a66a0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f31a89976b964b35a016585642a66a0f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39831 } }
I20260619 14:51:11.619256 21493 leader_election.cc:302] T 00000000000000000000000000000000 P f31a89976b964b35a016585642a66a0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f31a89976b964b35a016585642a66a0f; no voters:
I20260619 14:51:11.619690 21493 leader_election.cc:288] T 00000000000000000000000000000000 P f31a89976b964b35a016585642a66a0f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:51:11.619830 21502 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f31a89976b964b35a016585642a66a0f [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:51:11.620091 21502 raft_consensus.cc:686] T 00000000000000000000000000000000 P f31a89976b964b35a016585642a66a0f [term 1 LEADER]: Becoming Leader. State: Replica: f31a89976b964b35a016585642a66a0f, State: Running, Role: LEADER
I20260619 14:51:11.620465 21502 consensus_queue.cc:237] T 00000000000000000000000000000000 P f31a89976b964b35a016585642a66a0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f31a89976b964b35a016585642a66a0f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39831 } }
I20260619 14:51:11.621385 21493 sys_catalog.cc:564] T 00000000000000000000000000000000 P f31a89976b964b35a016585642a66a0f [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:51:11.622187 21504 sys_catalog.cc:455] T 00000000000000000000000000000000 P f31a89976b964b35a016585642a66a0f [sys.catalog]: SysCatalogTable state changed. Reason: New leader f31a89976b964b35a016585642a66a0f. Latest consensus state: current_term: 1 leader_uuid: "f31a89976b964b35a016585642a66a0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f31a89976b964b35a016585642a66a0f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39831 } } }
I20260619 14:51:11.622331 21504 sys_catalog.cc:458] T 00000000000000000000000000000000 P f31a89976b964b35a016585642a66a0f [sys.catalog]: This master's current role is: LEADER
I20260619 14:51:11.622655 21503 sys_catalog.cc:455] T 00000000000000000000000000000000 P f31a89976b964b35a016585642a66a0f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f31a89976b964b35a016585642a66a0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f31a89976b964b35a016585642a66a0f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39831 } } }
I20260619 14:51:11.622757 21503 sys_catalog.cc:458] T 00000000000000000000000000000000 P f31a89976b964b35a016585642a66a0f [sys.catalog]: This master's current role is: LEADER
I20260619 14:51:11.626267 21496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:11.620636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.164:43227 (local address 127.0.0.1:39831)
0619 14:51:11.620774 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:11.620779 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:11.620796 (+ 17us) server_negotiation.cc:407] Connection header received
0619 14:51:11.620836 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:11.620838 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:11.620889 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:11.620978 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:11.621374 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:11.622208 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:11.623578 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:11.624299 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:11.624428 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:11.625675 (+ 1247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:11.625679 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:11.625682 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:51:11.625700 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:11.625737 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:51:11.625750 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:11.625860 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:11.626041 (+ 181us) server_negotiation.cc:299] Negotiation successful
0619 14:51:11.626084 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
W20260619 14:51:11.627348 21519 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f31a89976b964b35a016585642a66a0f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:51:11.627522 21519 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:51:11.627607 21512 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:51:11.628577 21512 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:51:11.636014 21512 catalog_manager.cc:1347] Generated new cluster ID: 12428fd4c69b49dd95bfb6164605c85d
I20260619 14:51:11.636183 21512 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:51:11.668394 21512 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:51:11.669817 21512 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:51:11.676262 21512 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f31a89976b964b35a016585642a66a0f: Generated new TSK 0
I20260619 14:51:11.677227 21512 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:51:11.688542 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 21425
W20260619 14:51:11.693516 21410 connection.cc:535] client connection to 127.0.0.1:39831 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39831): Connection reset by peer (error 104)
2026-06-19T14:51:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1584 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781880383132139-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:51:11 dist-test-slave-2sb1 krb5kdc[21524](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:51:11 dist-test-slave-2sb1 krb5kdc[21524](info): set up 2 sockets
Jun 19 14:51:11 dist-test-slave-2sb1 krb5kdc[21524](info): commencing operation
krb5kdc: starting...
W20260619 14:51:12.810284 392 mini_kdc.cc:121] Time spent starting KDC: real 1.102s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781880383132139-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781880383132139-392-0/krb5kdc/test-user.keytab.
Jun 19 14:51:12 dist-test-slave-2sb1 krb5kdc[21524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880672, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:51:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:51:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781880383132139-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781880383132139-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 14:51:12.973253 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781880383132139-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781880383132139-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781880383132139-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34087
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781880383132139-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34087
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:51:13.115526 21537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:51:13.115957 21537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:51:13.116056 21537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:51:13.121227 21537 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:51:13.121340 21537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:51:13.121376 21537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:51:13.121417 21537 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:51:13.121459 21537 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:51:13.127492 21537 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781880383132139-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781880383132139-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34087
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781880383132139-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34087
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21537
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781880383132139-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:51:13.128775 21537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:51:13.130321 21537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:51:13.143287 21542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:51:13.143211 21545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:51:13.143211 21543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:51:13.143682 21537 server_base.cc:1034] running on GCE node
I20260619 14:51:13.144532 21537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:51:13.146085 21537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:51:13.147388 21537 hybrid_clock.cc:648] HybridClock initialized: now 1781880673147343 us; error 71 us; skew 500 ppm
Jun 19 14:51:13 dist-test-slave-2sb1 krb5kdc[21524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880673, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:51:13.151749 21537 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 14:51:13.153399 21537 webserver.cc:460] Webserver started at http://127.0.0.1:41229/ using document root <none> and password file <none>
I20260619 14:51:13.154114 21537 fs_manager.cc:359] Metadata directory not provided
I20260619 14:51:13.154237 21537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:51:13.154495 21537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:51:13.157003 21537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781880383132139-392-0/minicluster-data/master-0/data/instance:
uuid: "2e73e0957af44543931764e764a0c13e"
format_stamp: "Formatted at 2026-06-19 14:51:13 on dist-test-slave-2sb1"
server_key: "317d0f4ebc925a01dba58184637ee4e6"
server_key_iv: "2e4bd9048d0fb5d774d36ed90a3eba03"
server_key_version: "encryptionkey@0"
I20260619 14:51:13.157707 21537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781880383132139-392-0/minicluster-data/master-0/wal/instance:
uuid: "2e73e0957af44543931764e764a0c13e"
format_stamp: "Formatted at 2026-06-19 14:51:13 on dist-test-slave-2sb1"
server_key: "317d0f4ebc925a01dba58184637ee4e6"
server_key_iv: "2e4bd9048d0fb5d774d36ed90a3eba03"
server_key_version: "encryptionkey@0"
I20260619 14:51:13.162578 21537 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260619 14:51:13.166043 21552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:13.167857 21537 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260619 14:51:13.168068 21537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781880383132139-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781880383132139-392-0/minicluster-data/master-0/wal
uuid: "2e73e0957af44543931764e764a0c13e"
format_stamp: "Formatted at 2026-06-19 14:51:13 on dist-test-slave-2sb1"
server_key: "317d0f4ebc925a01dba58184637ee4e6"
server_key_iv: "2e4bd9048d0fb5d774d36ed90a3eba03"
server_key_version: "encryptionkey@0"
I20260619 14:51:13.168231 21537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781880383132139-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781880383132139-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781880383132139-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:51:13.174270 21537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:51:13.178205 21537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:51:13.178468 21537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:51:13.189551 21537 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34087
I20260619 14:51:13.189669 21604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34087 every 8 connection(s)
I20260619 14:51:13.190773 21537 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:51:13.191984 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 21537
I20260619 14:51:13.192173 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781880383132139-392-0/minicluster-data/master-0/wal/instance
I20260619 14:51:13.192504 392 external_mini_cluster.cc:1428] Setting key 1b57256496b8702bf18fabae4954cecc
I20260619 14:51:13.194773 21605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:51:13.199050 21608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:13.194158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.164:46191 (local address 127.0.0.1:34087)
0619 14:51:13.194775 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:13.194793 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:13.194835 (+ 42us) server_negotiation.cc:407] Connection header received
0619 14:51:13.195570 (+ 735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:13.195588 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:13.196003 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:13.196558 (+ 555us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:13.197586 (+ 1028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:13.197591 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:13.197603 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:51:13.197616 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:13.197673 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:51:13.197694 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:13.197856 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:13.197944 (+ 88us) server_negotiation.cc:299] Negotiation successful
0619 14:51:13.198265 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":190,"threads_started":1}
I20260619 14:51:13.203289 21605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e73e0957af44543931764e764a0c13e: Bootstrap starting.
I20260619 14:51:13.203691 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:51:13.207173 21605 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e73e0957af44543931764e764a0c13e: Neither blocks nor log segments found. Creating new log.
I20260619 14:51:13.208647 21605 log.cc:824] T 00000000000000000000000000000000 P 2e73e0957af44543931764e764a0c13e: Log is configured to *not* fsync() on all Append() calls
I20260619 14:51:13.209583 21608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:13.206583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.164:48697 (local address 127.0.0.1:34087)
0619 14:51:13.206713 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:13.206716 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:13.207076 (+ 360us) server_negotiation.cc:407] Connection header received
0619 14:51:13.207183 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:13.207186 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:13.207250 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:13.207342 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:13.209030 (+ 1688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:13.209033 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:13.209036 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:51:13.209052 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:13.209091 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:51:13.209106 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:13.209328 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:13.209395 (+ 67us) server_negotiation.cc:299] Negotiation successful
0619 14:51:13.209448 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 14:51:13.211531 21605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e73e0957af44543931764e764a0c13e: No bootstrap required, opened a new log
I20260619 14:51:13.215827 21605 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e73e0957af44543931764e764a0c13e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e73e0957af44543931764e764a0c13e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34087 } }
I20260619 14:51:13.216110 21605 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2e73e0957af44543931764e764a0c13e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:51:13.216156 21605 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e73e0957af44543931764e764a0c13e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e73e0957af44543931764e764a0c13e, State: Initialized, Role: FOLLOWER
I20260619 14:51:13.216746 21605 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e73e0957af44543931764e764a0c13e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e73e0957af44543931764e764a0c13e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34087 } }
I20260619 14:51:13.216888 21605 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2e73e0957af44543931764e764a0c13e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:51:13.216935 21605 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e73e0957af44543931764e764a0c13e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:51:13.217006 21605 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e73e0957af44543931764e764a0c13e [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:51:13.218286 21605 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e73e0957af44543931764e764a0c13e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e73e0957af44543931764e764a0c13e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34087 } }
I20260619 14:51:13.218674 21605 leader_election.cc:302] T 00000000000000000000000000000000 P 2e73e0957af44543931764e764a0c13e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e73e0957af44543931764e764a0c13e; no voters:
I20260619 14:51:13.219276 21605 leader_election.cc:288] T 00000000000000000000000000000000 P 2e73e0957af44543931764e764a0c13e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:51:13.219666 21615 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e73e0957af44543931764e764a0c13e [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:51:13.219895 21615 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2e73e0957af44543931764e764a0c13e [term 1 LEADER]: Becoming Leader. State: Replica: 2e73e0957af44543931764e764a0c13e, State: Running, Role: LEADER
I20260619 14:51:13.220256 21615 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e73e0957af44543931764e764a0c13e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e73e0957af44543931764e764a0c13e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34087 } }
I20260619 14:51:13.220863 21605 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e73e0957af44543931764e764a0c13e [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:51:13.222158 21616 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e73e0957af44543931764e764a0c13e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2e73e0957af44543931764e764a0c13e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e73e0957af44543931764e764a0c13e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34087 } } }
I20260619 14:51:13.222319 21616 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e73e0957af44543931764e764a0c13e [sys.catalog]: This master's current role is: LEADER
I20260619 14:51:13.222743 21624 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:51:13.222849 21617 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e73e0957af44543931764e764a0c13e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e73e0957af44543931764e764a0c13e. Latest consensus state: current_term: 1 leader_uuid: "2e73e0957af44543931764e764a0c13e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e73e0957af44543931764e764a0c13e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34087 } } }
I20260619 14:51:13.223054 21617 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e73e0957af44543931764e764a0c13e [sys.catalog]: This master's current role is: LEADER
I20260619 14:51:13.227360 21624 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:51:13.236860 21624 catalog_manager.cc:1347] Generated new cluster ID: 25de3f089d4b465a8f401dd2c9af9c79
I20260619 14:51:13.237147 21624 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:51:13.253350 21624 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:51:13.253837 21624 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:51:13.268445 21624 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2e73e0957af44543931764e764a0c13e: Generated new TSK 0
I20260619 14:51:13.269493 21624 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:51:13.282222 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 21537
2026-06-19T14:51:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1597 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781880383132139-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:51:13 dist-test-slave-2sb1 krb5kdc[21636](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:51:13 dist-test-slave-2sb1 krb5kdc[21636](info): set up 2 sockets
Jun 19 14:51:13 dist-test-slave-2sb1 krb5kdc[21636](info): commencing operation
krb5kdc: starting...
W20260619 14:51:14.421260 392 mini_kdc.cc:121] Time spent starting KDC: real 1.114s user 0.000s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781880383132139-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781880383132139-392-0/krb5kdc/test-user.keytab.
Jun 19 14:51:14 dist-test-slave-2sb1 krb5kdc[21636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880674, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:51:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:51:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781880383132139-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781880383132139-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 14:51:14.566663 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781880383132139-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781880383132139-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781880383132139-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44475
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781880383132139-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44475
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:51:14.696357 21649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:51:14.696784 21649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:51:14.696902 21649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:51:14.701694 21649 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:51:14.701807 21649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:51:14.701848 21649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:51:14.701908 21649 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:51:14.701948 21649 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:51:14.707968 21649 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781880383132139-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781880383132139-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44475
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781880383132139-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21649
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781880383132139-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:51:14.709318 21649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:51:14.710455 21649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:51:14.720165 21657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:51:14.720319 21654 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:51:14.720259 21649 server_base.cc:1034] running on GCE node
W20260619 14:51:14.720175 21655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:51:14.721500 21649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:51:14.722872 21649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:51:14.724090 21649 hybrid_clock.cc:648] HybridClock initialized: now 1781880674724104 us; error 132 us; skew 500 ppm
Jun 19 14:51:14 dist-test-slave-2sb1 krb5kdc[21636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880674, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:51:14.727473 21649 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 14:51:14.728564 21649 webserver.cc:460] Webserver started at http://127.0.0.1:43747/ using document root <none> and password file <none>
I20260619 14:51:14.729189 21649 fs_manager.cc:359] Metadata directory not provided
I20260619 14:51:14.729287 21649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:51:14.729547 21649 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:51:14.731689 21649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781880383132139-392-0/minicluster-data/master-0/data/instance:
uuid: "85f35316ad99414a9bd987d2def7e5c6"
format_stamp: "Formatted at 2026-06-19 14:51:14 on dist-test-slave-2sb1"
server_key: "f6f281c4d8c1b956277bd6a1fd1f82c5"
server_key_iv: "92d2e69a2f3724f13eab8e50c172b3ab"
server_key_version: "encryptionkey@0"
I20260619 14:51:14.732296 21649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781880383132139-392-0/minicluster-data/master-0/wal/instance:
uuid: "85f35316ad99414a9bd987d2def7e5c6"
format_stamp: "Formatted at 2026-06-19 14:51:14 on dist-test-slave-2sb1"
server_key: "f6f281c4d8c1b956277bd6a1fd1f82c5"
server_key_iv: "92d2e69a2f3724f13eab8e50c172b3ab"
server_key_version: "encryptionkey@0"
I20260619 14:51:14.736614 21649 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260619 14:51:14.739609 21664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:14.741351 21649 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:51:14.741514 21649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781880383132139-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781880383132139-392-0/minicluster-data/master-0/wal
uuid: "85f35316ad99414a9bd987d2def7e5c6"
format_stamp: "Formatted at 2026-06-19 14:51:14 on dist-test-slave-2sb1"
server_key: "f6f281c4d8c1b956277bd6a1fd1f82c5"
server_key_iv: "92d2e69a2f3724f13eab8e50c172b3ab"
server_key_version: "encryptionkey@0"
I20260619 14:51:14.741658 21649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781880383132139-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781880383132139-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781880383132139-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:51:14.770391 21649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:51:14.773993 21649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:51:14.774271 21649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:51:14.782114 21649 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44475
I20260619 14:51:14.782141 21716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44475 every 8 connection(s)
I20260619 14:51:14.783212 21649 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:51:14.784704 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 21649
I20260619 14:51:14.784857 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781880383132139-392-0/minicluster-data/master-0/wal/instance
I20260619 14:51:14.785257 392 external_mini_cluster.cc:1428] Setting key dcd8abeef2eb937c0d51fc8bd735a8ef
I20260619 14:51:14.786352 21717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:51:14.792287 21717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85f35316ad99414a9bd987d2def7e5c6: Bootstrap starting.
Jun 19 14:51:14 dist-test-slave-2sb1 krb5kdc[21636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880674, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260619 14:51:14.795073 21717 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 85f35316ad99414a9bd987d2def7e5c6: Neither blocks nor log segments found. Creating new log.
I20260619 14:51:14.796001 21717 log.cc:824] T 00000000000000000000000000000000 P 85f35316ad99414a9bd987d2def7e5c6: Log is configured to *not* fsync() on all Append() calls
I20260619 14:51:14.798640 21717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85f35316ad99414a9bd987d2def7e5c6: No bootstrap required, opened a new log
I20260619 14:51:14.801738 21720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:14.786737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.164:55535 (local address 127.0.0.1:44475)
0619 14:51:14.787299 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:14.787316 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:14.787353 (+ 37us) server_negotiation.cc:407] Connection header received
0619 14:51:14.788004 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:14.788021 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:14.788390 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:14.788782 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:14.789628 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:14.790640 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:14.791552 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:14.792045 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:14.792244 (+ 199us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:14.795370 (+ 3126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:14.795377 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:14.795394 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:14.795429 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:14.797864 (+ 2435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:14.798621 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:14.798627 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:14.798633 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:14.798815 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:14.799231 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:14.799234 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:14.799236 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:14.799773 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:14.799995 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:14.800333 (+ 338us) server_negotiation.cc:299] Negotiation successful
0619 14:51:14.800577 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":149,"threads_started":1}
I20260619 14:51:14.803449 21717 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85f35316ad99414a9bd987d2def7e5c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85f35316ad99414a9bd987d2def7e5c6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44475 } }
I20260619 14:51:14.803727 21717 raft_consensus.cc:374] T 00000000000000000000000000000000 P 85f35316ad99414a9bd987d2def7e5c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:51:14.803792 21717 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85f35316ad99414a9bd987d2def7e5c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85f35316ad99414a9bd987d2def7e5c6, State: Initialized, Role: FOLLOWER
I20260619 14:51:14.804231 21717 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85f35316ad99414a9bd987d2def7e5c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85f35316ad99414a9bd987d2def7e5c6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44475 } }
I20260619 14:51:14.804363 21717 raft_consensus.cc:388] T 00000000000000000000000000000000 P 85f35316ad99414a9bd987d2def7e5c6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:51:14.804452 21717 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85f35316ad99414a9bd987d2def7e5c6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:51:14.804565 21717 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85f35316ad99414a9bd987d2def7e5c6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:51:14.805239 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:51:14.805760 21717 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85f35316ad99414a9bd987d2def7e5c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85f35316ad99414a9bd987d2def7e5c6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44475 } }
I20260619 14:51:14.806128 21717 leader_election.cc:302] T 00000000000000000000000000000000 P 85f35316ad99414a9bd987d2def7e5c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85f35316ad99414a9bd987d2def7e5c6; no voters:
I20260619 14:51:14.806493 21717 leader_election.cc:288] T 00000000000000000000000000000000 P 85f35316ad99414a9bd987d2def7e5c6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:51:14.806650 21723 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 85f35316ad99414a9bd987d2def7e5c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:51:14.807032 21723 raft_consensus.cc:686] T 00000000000000000000000000000000 P 85f35316ad99414a9bd987d2def7e5c6 [term 1 LEADER]: Becoming Leader. State: Replica: 85f35316ad99414a9bd987d2def7e5c6, State: Running, Role: LEADER
I20260619 14:51:14.807518 21723 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85f35316ad99414a9bd987d2def7e5c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85f35316ad99414a9bd987d2def7e5c6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44475 } }
I20260619 14:51:14.808023 21717 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85f35316ad99414a9bd987d2def7e5c6 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:51:14.809442 21727 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85f35316ad99414a9bd987d2def7e5c6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "85f35316ad99414a9bd987d2def7e5c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85f35316ad99414a9bd987d2def7e5c6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44475 } } }
I20260619 14:51:14.809584 21727 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85f35316ad99414a9bd987d2def7e5c6 [sys.catalog]: This master's current role is: LEADER
I20260619 14:51:14.809876 21728 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85f35316ad99414a9bd987d2def7e5c6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85f35316ad99414a9bd987d2def7e5c6. Latest consensus state: current_term: 1 leader_uuid: "85f35316ad99414a9bd987d2def7e5c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85f35316ad99414a9bd987d2def7e5c6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44475 } } }
I20260619 14:51:14.809957 21728 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85f35316ad99414a9bd987d2def7e5c6 [sys.catalog]: This master's current role is: LEADER
I20260619 14:51:14.810305 21738 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:51:14.814085 21738 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:51:14.817710 21720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:14.807504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.164:49545 (local address 127.0.0.1:44475)
0619 14:51:14.807642 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:14.807647 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:14.807929 (+ 282us) server_negotiation.cc:407] Connection header received
0619 14:51:14.808054 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:14.808057 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:14.808111 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:14.808199 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:14.809275 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:14.810098 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:14.811344 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:14.811672 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:14.811797 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:14.813665 (+ 1868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:14.813669 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:14.813672 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:14.813701 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:14.815519 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:14.816196 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:14.816199 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:14.816201 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:14.816294 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:14.816702 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:14.816705 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:14.816707 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:14.816927 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:14.817088 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:14.817451 (+ 363us) server_negotiation.cc:299] Negotiation successful
0619 14:51:14.817555 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 14:51:14.820627 21738 catalog_manager.cc:1347] Generated new cluster ID: ce170e86c8fb494bb432b0b4f83434c5
I20260619 14:51:14.820771 21738 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:51:14.850595 21738 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:51:14.851678 21738 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:51:14.857942 21738 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 85f35316ad99414a9bd987d2def7e5c6: Generated new TSK 0
I20260619 14:51:14.858873 21738 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:51:14.867965 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 21649
W20260619 14:51:14.872866 21634 connection.cc:535] client connection to 127.0.0.1:44475 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44475): Connection reset by peer (error 104)
2026-06-19T14:51:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1581 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781880383132139-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:51:14 dist-test-slave-2sb1 krb5kdc[21748](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:51:14 dist-test-slave-2sb1 krb5kdc[21748](info): set up 2 sockets
Jun 19 14:51:14 dist-test-slave-2sb1 krb5kdc[21748](info): commencing operation
krb5kdc: starting...
W20260619 14:51:15.937106 392 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781880383132139-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781880383132139-392-0/krb5kdc/test-user.keytab.
Jun 19 14:51:16 dist-test-slave-2sb1 krb5kdc[21748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880676, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:51:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:51:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781880383132139-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781880383132139-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 14:51:16.076839 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781880383132139-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781880383132139-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781880383132139-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45225
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781880383132139-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45225
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:51:16.197001 21761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:51:16.197325 21761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:51:16.197371 21761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:51:16.201957 21761 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:51:16.202037 21761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:51:16.202059 21761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:51:16.202075 21761 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:51:16.202095 21761 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:51:16.207911 21761 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781880383132139-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781880383132139-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45225
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781880383132139-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21761
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781880383132139-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:51:16.209152 21761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:51:16.210253 21761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:51:16.219962 21767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:51:16.219998 21769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:51:16.220283 21766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:51:16.220429 21761 server_base.cc:1034] running on GCE node
I20260619 14:51:16.221248 21761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:51:16.222465 21761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:51:16.223695 21761 hybrid_clock.cc:648] HybridClock initialized: now 1781880676223669 us; error 44 us; skew 500 ppm
Jun 19 14:51:16 dist-test-slave-2sb1 krb5kdc[21748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880676, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:51:16.234268 21761 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 14:51:16.235514 21761 webserver.cc:460] Webserver started at http://127.0.0.1:37085/ using document root <none> and password file <none>
I20260619 14:51:16.236178 21761 fs_manager.cc:359] Metadata directory not provided
I20260619 14:51:16.236282 21761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:51:16.236531 21761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:51:16.238859 21761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781880383132139-392-0/minicluster-data/master-0/data/instance:
uuid: "7271e12760f34de88e5e9323281a7720"
format_stamp: "Formatted at 2026-06-19 14:51:16 on dist-test-slave-2sb1"
server_key: "e3107d55f505443fe7cd401ba319dafb"
server_key_iv: "a4101593b51bc20a97b24985159e8ca5"
server_key_version: "encryptionkey@0"
I20260619 14:51:16.239511 21761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781880383132139-392-0/minicluster-data/master-0/wal/instance:
uuid: "7271e12760f34de88e5e9323281a7720"
format_stamp: "Formatted at 2026-06-19 14:51:16 on dist-test-slave-2sb1"
server_key: "e3107d55f505443fe7cd401ba319dafb"
server_key_iv: "a4101593b51bc20a97b24985159e8ca5"
server_key_version: "encryptionkey@0"
I20260619 14:51:16.244191 21761 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 14:51:16.247406 21776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:16.249034 21761 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:51:16.249221 21761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781880383132139-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781880383132139-392-0/minicluster-data/master-0/wal
uuid: "7271e12760f34de88e5e9323281a7720"
format_stamp: "Formatted at 2026-06-19 14:51:16 on dist-test-slave-2sb1"
server_key: "e3107d55f505443fe7cd401ba319dafb"
server_key_iv: "a4101593b51bc20a97b24985159e8ca5"
server_key_version: "encryptionkey@0"
I20260619 14:51:16.249366 21761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781880383132139-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781880383132139-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781880383132139-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:51:16.270814 21761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:51:16.275154 21761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:51:16.275398 21761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:51:16.284502 21761 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45225
I20260619 14:51:16.284531 21828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45225 every 8 connection(s)
I20260619 14:51:16.285583 21761 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:51:16.288841 21829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:51:16.294754 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 21761
I20260619 14:51:16.294945 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781880383132139-392-0/minicluster-data/master-0/wal/instance
I20260619 14:51:16.295037 21829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7271e12760f34de88e5e9323281a7720: Bootstrap starting.
I20260619 14:51:16.295343 392 external_mini_cluster.cc:1428] Setting key c93a577fdf2f6e15cde76a318933f0d1
I20260619 14:51:16.297871 21829 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7271e12760f34de88e5e9323281a7720: Neither blocks nor log segments found. Creating new log.
I20260619 14:51:16.298699 21829 log.cc:824] T 00000000000000000000000000000000 P 7271e12760f34de88e5e9323281a7720: Log is configured to *not* fsync() on all Append() calls
I20260619 14:51:16.300770 21829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7271e12760f34de88e5e9323281a7720: No bootstrap required, opened a new log
I20260619 14:51:16.305020 21829 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7271e12760f34de88e5e9323281a7720 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7271e12760f34de88e5e9323281a7720" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45225 } }
I20260619 14:51:16.305351 21829 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7271e12760f34de88e5e9323281a7720 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:51:16.305400 21829 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7271e12760f34de88e5e9323281a7720 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7271e12760f34de88e5e9323281a7720, State: Initialized, Role: FOLLOWER
I20260619 14:51:16.305784 21832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:16.296777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.164:49217 (local address 127.0.0.1:45225)
0619 14:51:16.297335 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:16.297352 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:16.297399 (+ 47us) server_negotiation.cc:407] Connection header received
0619 14:51:16.298219 (+ 820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:16.298241 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:16.298645 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:16.299059 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:16.299601 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:16.300563 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:16.301647 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:16.302147 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:16.302350 (+ 203us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:16.303431 (+ 1081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:16.303439 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:16.303461 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:51:16.304040 (+ 579us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:16.304123 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:51:16.304156 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:16.304312 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:16.304512 (+ 200us) server_negotiation.cc:299] Negotiation successful
0619 14:51:16.304696 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":195,"threads_started":1}
I20260619 14:51:16.305882 21829 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7271e12760f34de88e5e9323281a7720 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7271e12760f34de88e5e9323281a7720" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45225 } }
I20260619 14:51:16.306039 21829 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7271e12760f34de88e5e9323281a7720 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:51:16.306121 21829 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7271e12760f34de88e5e9323281a7720 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:51:16.306210 21829 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7271e12760f34de88e5e9323281a7720 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:51:16.307380 21829 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7271e12760f34de88e5e9323281a7720 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7271e12760f34de88e5e9323281a7720" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45225 } }
I20260619 14:51:16.307740 21829 leader_election.cc:302] T 00000000000000000000000000000000 P 7271e12760f34de88e5e9323281a7720 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7271e12760f34de88e5e9323281a7720; no voters:
I20260619 14:51:16.308156 21829 leader_election.cc:288] T 00000000000000000000000000000000 P 7271e12760f34de88e5e9323281a7720 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:51:16.308297 21834 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7271e12760f34de88e5e9323281a7720 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:51:16.308570 21834 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7271e12760f34de88e5e9323281a7720 [term 1 LEADER]: Becoming Leader. State: Replica: 7271e12760f34de88e5e9323281a7720, State: Running, Role: LEADER
I20260619 14:51:16.309442 21834 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7271e12760f34de88e5e9323281a7720 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7271e12760f34de88e5e9323281a7720" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45225 } }
I20260619 14:51:16.309563 21829 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7271e12760f34de88e5e9323281a7720 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:51:16.309697 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:51:16.311486 21835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7271e12760f34de88e5e9323281a7720 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7271e12760f34de88e5e9323281a7720" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7271e12760f34de88e5e9323281a7720" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45225 } } }
I20260619 14:51:16.311472 21836 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7271e12760f34de88e5e9323281a7720 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7271e12760f34de88e5e9323281a7720. Latest consensus state: current_term: 1 leader_uuid: "7271e12760f34de88e5e9323281a7720" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7271e12760f34de88e5e9323281a7720" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45225 } } }
I20260619 14:51:16.311656 21836 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7271e12760f34de88e5e9323281a7720 [sys.catalog]: This master's current role is: LEADER
I20260619 14:51:16.311656 21835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7271e12760f34de88e5e9323281a7720 [sys.catalog]: This master's current role is: LEADER
W20260619 14:51:16.315698 21852 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7271e12760f34de88e5e9323281a7720: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:51:16.315833 21852 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:51:16.315943 21853 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:51:16.317241 21853 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:51:16.317425 21832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:16.311999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.164:49463 (local address 127.0.0.1:45225)
0619 14:51:16.312140 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:16.312145 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:16.312620 (+ 475us) server_negotiation.cc:407] Connection header received
0619 14:51:16.312812 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:16.312815 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:16.312868 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:16.312919 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:16.313297 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:16.313991 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:16.315190 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:16.315518 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:16.315637 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:16.316876 (+ 1239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:16.316880 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:16.316883 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:51:16.316900 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:16.316930 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:51:16.316944 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:16.317052 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:16.317234 (+ 182us) server_negotiation.cc:299] Negotiation successful
0619 14:51:16.317278 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260619 14:51:16.324343 21853 catalog_manager.cc:1347] Generated new cluster ID: 8b65fa688f3b40fca99c70b8ae2a4c32
I20260619 14:51:16.324443 21853 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:51:16.353300 21853 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:51:16.354524 21853 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:51:16.364095 21853 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7271e12760f34de88e5e9323281a7720: Generated new TSK 0
I20260619 14:51:16.364946 21853 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:51:16.379917 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 21761
W20260619 14:51:16.387930 21746 connection.cc:535] client connection to 127.0.0.1:45225 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45225): Connection reset by peer (error 104)
2026-06-19T14:51:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1515 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781880383132139-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:51:16 dist-test-slave-2sb1 krb5kdc[21860](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:51:16 dist-test-slave-2sb1 krb5kdc[21860](info): set up 2 sockets
Jun 19 14:51:16 dist-test-slave-2sb1 krb5kdc[21860](info): commencing operation
krb5kdc: starting...
W20260619 14:51:17.469620 392 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781880383132139-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781880383132139-392-0/krb5kdc/test-user.keytab.
Jun 19 14:51:17 dist-test-slave-2sb1 krb5kdc[21860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880677, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:51:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:51:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781880383132139-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781880383132139-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260619 14:51:17.613658 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781880383132139-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781880383132139-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781880383132139-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37781
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781880383132139-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37781
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:51:17.732897 21873 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:51:17.733242 21873 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:51:17.733286 21873 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:51:17.737664 21873 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:51:17.737793 21873 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:51:17.737818 21873 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:51:17.737835 21873 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:51:17.737850 21873 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:51:17.743853 21873 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781880383132139-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781880383132139-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37781
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781880383132139-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21873
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781880383132139-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:51:17.745082 21873 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:51:17.746116 21873 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 14:51:17.756314 21873 server_base.cc:1034] running on GCE node
W20260619 14:51:17.756502 21878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:51:17.756263 21881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:51:17.756224 21879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:51:17.757293 21873 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:51:17.758607 21873 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:51:17.759845 21873 hybrid_clock.cc:648] HybridClock initialized: now 1781880677759736 us; error 127 us; skew 500 ppm
Jun 19 14:51:17 dist-test-slave-2sb1 krb5kdc[21860](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880677, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:51:17.763518 21873 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260619 14:51:17.764618 21873 webserver.cc:460] Webserver started at http://127.0.0.1:46049/ using document root <none> and password file <none>
I20260619 14:51:17.765491 21873 fs_manager.cc:359] Metadata directory not provided
I20260619 14:51:17.765602 21873 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:51:17.765864 21873 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:51:17.768307 21873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781880383132139-392-0/minicluster-data/master-0/data/instance:
uuid: "d6d39e477d0f41e2b8700e032628dd35"
format_stamp: "Formatted at 2026-06-19 14:51:17 on dist-test-slave-2sb1"
server_key: "d5bad6a8f8d65210ecb2e86fba96587a"
server_key_iv: "3bcc36821ddeabd19a87856be27e8983"
server_key_version: "encryptionkey@0"
I20260619 14:51:17.769068 21873 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781880383132139-392-0/minicluster-data/master-0/wal/instance:
uuid: "d6d39e477d0f41e2b8700e032628dd35"
format_stamp: "Formatted at 2026-06-19 14:51:17 on dist-test-slave-2sb1"
server_key: "d5bad6a8f8d65210ecb2e86fba96587a"
server_key_iv: "3bcc36821ddeabd19a87856be27e8983"
server_key_version: "encryptionkey@0"
I20260619 14:51:17.773752 21873 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 14:51:17.776582 21888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:17.778281 21873 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 14:51:17.778476 21873 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781880383132139-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781880383132139-392-0/minicluster-data/master-0/wal
uuid: "d6d39e477d0f41e2b8700e032628dd35"
format_stamp: "Formatted at 2026-06-19 14:51:17 on dist-test-slave-2sb1"
server_key: "d5bad6a8f8d65210ecb2e86fba96587a"
server_key_iv: "3bcc36821ddeabd19a87856be27e8983"
server_key_version: "encryptionkey@0"
I20260619 14:51:17.778625 21873 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781880383132139-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781880383132139-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781880383132139-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:51:17.784036 21873 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:51:17.787748 21873 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:51:17.787988 21873 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:51:17.795521 21873 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37781
I20260619 14:51:17.795567 21940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37781 every 8 connection(s)
I20260619 14:51:17.796806 21873 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:51:17.800117 21941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:51:17.801643 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 21873
I20260619 14:51:17.801796 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781880383132139-392-0/minicluster-data/master-0/wal/instance
I20260619 14:51:17.802070 392 external_mini_cluster.cc:1428] Setting key ff90fc82d2fc783ac698c24590bc7250
I20260619 14:51:17.808377 21941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6d39e477d0f41e2b8700e032628dd35: Bootstrap starting.
I20260619 14:51:17.808981 21944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:17.803785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.164:45707 (local address 127.0.0.1:37781)
0619 14:51:17.804308 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:17.804329 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:17.804377 (+ 48us) server_negotiation.cc:407] Connection header received
0619 14:51:17.805328 (+ 951us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:17.805356 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:17.805777 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:17.806225 (+ 448us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:17.807328 (+ 1103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:17.807335 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:17.807352 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:51:17.807366 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:17.807443 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:51:17.807475 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:17.807640 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:17.807865 (+ 225us) server_negotiation.cc:299] Negotiation successful
0619 14:51:17.808061 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":132,"threads_started":1}
I20260619 14:51:17.811061 21941 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d6d39e477d0f41e2b8700e032628dd35: Neither blocks nor log segments found. Creating new log.
I20260619 14:51:17.812126 21941 log.cc:824] T 00000000000000000000000000000000 P d6d39e477d0f41e2b8700e032628dd35: Log is configured to *not* fsync() on all Append() calls
I20260619 14:51:17.812219 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:51:17.814697 21941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6d39e477d0f41e2b8700e032628dd35: No bootstrap required, opened a new log
I20260619 14:51:17.816390 21944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:17.814413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.164:35233 (local address 127.0.0.1:37781)
0619 14:51:17.814568 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:17.814573 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:17.814847 (+ 274us) server_negotiation.cc:407] Connection header received
0619 14:51:17.814971 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:17.814974 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:17.815006 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:17.815071 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:17.815868 (+ 797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:17.815871 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:17.815873 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:51:17.815884 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:17.815911 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:51:17.815921 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:17.816006 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:17.816151 (+ 145us) server_negotiation.cc:299] Negotiation successful
0619 14:51:17.816229 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260619 14:51:17.819296 21941 raft_consensus.cc:348] T 00000000000000000000000000000000 P d6d39e477d0f41e2b8700e032628dd35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6d39e477d0f41e2b8700e032628dd35" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37781 } }
I20260619 14:51:17.819590 21941 raft_consensus.cc:374] T 00000000000000000000000000000000 P d6d39e477d0f41e2b8700e032628dd35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:51:17.819628 21941 raft_consensus.cc:729] T 00000000000000000000000000000000 P d6d39e477d0f41e2b8700e032628dd35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6d39e477d0f41e2b8700e032628dd35, State: Initialized, Role: FOLLOWER
I20260619 14:51:17.820118 21941 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6d39e477d0f41e2b8700e032628dd35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6d39e477d0f41e2b8700e032628dd35" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37781 } }
I20260619 14:51:17.820261 21941 raft_consensus.cc:388] T 00000000000000000000000000000000 P d6d39e477d0f41e2b8700e032628dd35 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:51:17.820307 21941 raft_consensus.cc:482] T 00000000000000000000000000000000 P d6d39e477d0f41e2b8700e032628dd35 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:51:17.820380 21941 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d6d39e477d0f41e2b8700e032628dd35 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:51:17.821583 21941 raft_consensus.cc:504] T 00000000000000000000000000000000 P d6d39e477d0f41e2b8700e032628dd35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6d39e477d0f41e2b8700e032628dd35" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37781 } }
I20260619 14:51:17.822000 21941 leader_election.cc:302] T 00000000000000000000000000000000 P d6d39e477d0f41e2b8700e032628dd35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d6d39e477d0f41e2b8700e032628dd35; no voters:
I20260619 14:51:17.822307 21941 leader_election.cc:288] T 00000000000000000000000000000000 P d6d39e477d0f41e2b8700e032628dd35 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:51:17.822453 21951 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d6d39e477d0f41e2b8700e032628dd35 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:51:17.822760 21951 raft_consensus.cc:686] T 00000000000000000000000000000000 P d6d39e477d0f41e2b8700e032628dd35 [term 1 LEADER]: Becoming Leader. State: Replica: d6d39e477d0f41e2b8700e032628dd35, State: Running, Role: LEADER
I20260619 14:51:17.823119 21951 consensus_queue.cc:237] T 00000000000000000000000000000000 P d6d39e477d0f41e2b8700e032628dd35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6d39e477d0f41e2b8700e032628dd35" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37781 } }
I20260619 14:51:17.823678 21941 sys_catalog.cc:564] T 00000000000000000000000000000000 P d6d39e477d0f41e2b8700e032628dd35 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:51:17.825026 21953 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6d39e477d0f41e2b8700e032628dd35 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6d39e477d0f41e2b8700e032628dd35. Latest consensus state: current_term: 1 leader_uuid: "d6d39e477d0f41e2b8700e032628dd35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6d39e477d0f41e2b8700e032628dd35" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37781 } } }
I20260619 14:51:17.825157 21953 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6d39e477d0f41e2b8700e032628dd35 [sys.catalog]: This master's current role is: LEADER
I20260619 14:51:17.825579 21952 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6d39e477d0f41e2b8700e032628dd35 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d6d39e477d0f41e2b8700e032628dd35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6d39e477d0f41e2b8700e032628dd35" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37781 } } }
I20260619 14:51:17.825736 21952 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6d39e477d0f41e2b8700e032628dd35 [sys.catalog]: This master's current role is: LEADER
W20260619 14:51:17.828447 21966 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d6d39e477d0f41e2b8700e032628dd35: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:51:17.828574 21966 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:51:17.828701 21960 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:51:17.829432 21960 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:51:17.836397 21960 catalog_manager.cc:1347] Generated new cluster ID: 3c9c1f25097b44d0b0c639736f1d4b10
I20260619 14:51:17.836499 21960 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:51:17.862948 21960 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:51:17.863350 21960 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:51:17.870612 21960 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d6d39e477d0f41e2b8700e032628dd35: Generated new TSK 0
I20260619 14:51:17.871575 21960 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:51:17.883476 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 21873
2026-06-19T14:51:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1501 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781880383132139-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:51:17 dist-test-slave-2sb1 krb5kdc[21972](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:51:17 dist-test-slave-2sb1 krb5kdc[21972](info): set up 2 sockets
Jun 19 14:51:17 dist-test-slave-2sb1 krb5kdc[21972](info): commencing operation
krb5kdc: starting...
W20260619 14:51:18.981189 392 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781880383132139-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781880383132139-392-0/krb5kdc/test-user.keytab.
Jun 19 14:51:19 dist-test-slave-2sb1 krb5kdc[21972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:51:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:51:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 14:51:19.129967 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781880383132139-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781880383132139-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781880383132139-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42477
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42477
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:51:19.260331 21985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:51:19.260771 21985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:51:19.260829 21985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:51:19.265259 21985 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:51:19.265347 21985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:51:19.265368 21985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:51:19.265388 21985 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:51:19.265403 21985 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:51:19.271531 21985 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781880383132139-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781880383132139-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42477
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42477
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21985
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781880383132139-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:51:19.272728 21985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:51:19.273772 21985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:51:19.283782 21990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:51:19.283859 21991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:51:19.283797 21993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:51:19.283833 21985 server_base.cc:1034] running on GCE node
I20260619 14:51:19.285158 21985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:51:19.286430 21985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:51:19.287683 21985 hybrid_clock.cc:648] HybridClock initialized: now 1781880679287645 us; error 59 us; skew 500 ppm
Jun 19 14:51:19 dist-test-slave-2sb1 krb5kdc[21972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880679, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:51:19.291824 21985 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 14:51:19.293073 21985 webserver.cc:460] Webserver started at http://127.0.98.62:39191/ using document root <none> and password file <none>
I20260619 14:51:19.293969 21985 fs_manager.cc:359] Metadata directory not provided
I20260619 14:51:19.294104 21985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:51:19.294384 21985 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:51:19.297348 21985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781880383132139-392-0/minicluster-data/master-0/data/instance:
uuid: "32c2516788e24d319e6ad1a6f0acbeb5"
format_stamp: "Formatted at 2026-06-19 14:51:19 on dist-test-slave-2sb1"
server_key: "d506c40b1eda1a8430c7c157def5840d"
server_key_iv: "40bf5f0fb8d09cba6cd5fce8a2d6d74c"
server_key_version: "encryptionkey@0"
I20260619 14:51:19.298275 21985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781880383132139-392-0/minicluster-data/master-0/wal/instance:
uuid: "32c2516788e24d319e6ad1a6f0acbeb5"
format_stamp: "Formatted at 2026-06-19 14:51:19 on dist-test-slave-2sb1"
server_key: "d506c40b1eda1a8430c7c157def5840d"
server_key_iv: "40bf5f0fb8d09cba6cd5fce8a2d6d74c"
server_key_version: "encryptionkey@0"
I20260619 14:51:19.303833 21985 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 14:51:19.308336 22000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:19.310288 21985 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260619 14:51:19.310513 21985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781880383132139-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781880383132139-392-0/minicluster-data/master-0/wal
uuid: "32c2516788e24d319e6ad1a6f0acbeb5"
format_stamp: "Formatted at 2026-06-19 14:51:19 on dist-test-slave-2sb1"
server_key: "d506c40b1eda1a8430c7c157def5840d"
server_key_iv: "40bf5f0fb8d09cba6cd5fce8a2d6d74c"
server_key_version: "encryptionkey@0"
I20260619 14:51:19.310684 21985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781880383132139-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781880383132139-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781880383132139-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:51:19.324056 21985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:51:19.330116 21985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:51:19.330400 21985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:51:19.338505 21985 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42477
I20260619 14:51:19.338568 22052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42477 every 8 connection(s)
I20260619 14:51:19.339632 21985 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:51:19.342993 22053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:51:19.348826 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 21985
I20260619 14:51:19.348989 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781880383132139-392-0/minicluster-data/master-0/wal/instance
I20260619 14:51:19.349278 392 external_mini_cluster.cc:1428] Setting key ff2cee2134f030ae1aedeb7df4dfae27
I20260619 14:51:19.349473 22053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32c2516788e24d319e6ad1a6f0acbeb5: Bootstrap starting.
I20260619 14:51:19.352114 22053 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 32c2516788e24d319e6ad1a6f0acbeb5: Neither blocks nor log segments found. Creating new log.
I20260619 14:51:19.353062 22053 log.cc:824] T 00000000000000000000000000000000 P 32c2516788e24d319e6ad1a6f0acbeb5: Log is configured to *not* fsync() on all Append() calls
I20260619 14:51:19.355894 22053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32c2516788e24d319e6ad1a6f0acbeb5: No bootstrap required, opened a new log
Jun 19 14:51:19 dist-test-slave-2sb1 krb5kdc[21972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:51:19.361030 22053 raft_consensus.cc:348] T 00000000000000000000000000000000 P 32c2516788e24d319e6ad1a6f0acbeb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32c2516788e24d319e6ad1a6f0acbeb5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42477 } }
I20260619 14:51:19.361409 22053 raft_consensus.cc:374] T 00000000000000000000000000000000 P 32c2516788e24d319e6ad1a6f0acbeb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:51:19.361577 22053 raft_consensus.cc:729] T 00000000000000000000000000000000 P 32c2516788e24d319e6ad1a6f0acbeb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32c2516788e24d319e6ad1a6f0acbeb5, State: Initialized, Role: FOLLOWER
I20260619 14:51:19.362178 22053 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32c2516788e24d319e6ad1a6f0acbeb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32c2516788e24d319e6ad1a6f0acbeb5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42477 } }
I20260619 14:51:19.362324 22053 raft_consensus.cc:388] T 00000000000000000000000000000000 P 32c2516788e24d319e6ad1a6f0acbeb5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:51:19.362422 22053 raft_consensus.cc:482] T 00000000000000000000000000000000 P 32c2516788e24d319e6ad1a6f0acbeb5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:51:19.362555 22053 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 32c2516788e24d319e6ad1a6f0acbeb5 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:51:19.363606 22053 raft_consensus.cc:504] T 00000000000000000000000000000000 P 32c2516788e24d319e6ad1a6f0acbeb5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32c2516788e24d319e6ad1a6f0acbeb5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42477 } }
I20260619 14:51:19.363989 22053 leader_election.cc:302] T 00000000000000000000000000000000 P 32c2516788e24d319e6ad1a6f0acbeb5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 32c2516788e24d319e6ad1a6f0acbeb5; no voters:
I20260619 14:51:19.364405 22053 leader_election.cc:288] T 00000000000000000000000000000000 P 32c2516788e24d319e6ad1a6f0acbeb5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:51:19.364519 22058 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 32c2516788e24d319e6ad1a6f0acbeb5 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:51:19.364799 22058 raft_consensus.cc:686] T 00000000000000000000000000000000 P 32c2516788e24d319e6ad1a6f0acbeb5 [term 1 LEADER]: Becoming Leader. State: Replica: 32c2516788e24d319e6ad1a6f0acbeb5, State: Running, Role: LEADER
I20260619 14:51:19.365206 22058 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32c2516788e24d319e6ad1a6f0acbeb5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32c2516788e24d319e6ad1a6f0acbeb5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42477 } }
I20260619 14:51:19.365913 22053 sys_catalog.cc:564] T 00000000000000000000000000000000 P 32c2516788e24d319e6ad1a6f0acbeb5 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:51:19.366631 22058 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32c2516788e24d319e6ad1a6f0acbeb5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32c2516788e24d319e6ad1a6f0acbeb5. Latest consensus state: current_term: 1 leader_uuid: "32c2516788e24d319e6ad1a6f0acbeb5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32c2516788e24d319e6ad1a6f0acbeb5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42477 } } }
I20260619 14:51:19.366809 22058 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32c2516788e24d319e6ad1a6f0acbeb5 [sys.catalog]: This master's current role is: LEADER
I20260619 14:51:19.366814 22056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:19.350877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59059 (local address 127.0.98.62:42477)
0619 14:51:19.351352 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:19.351377 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:19.351443 (+ 66us) server_negotiation.cc:407] Connection header received
0619 14:51:19.352153 (+ 710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:19.352206 (+ 53us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:19.352617 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:19.352991 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:19.353973 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:19.355056 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:19.356355 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:19.357003 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:19.357170 (+ 167us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:19.360576 (+ 3406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:19.360583 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:19.360600 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:19.360638 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:19.363122 (+ 2484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:19.363736 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:19.363744 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:19.363752 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:19.363895 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:19.364286 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:19.364292 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:19.364296 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:19.364943 (+ 647us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:19.365161 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:19.365527 (+ 366us) server_negotiation.cc:299] Negotiation successful
0619 14:51:19.365787 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":156,"threads_started":1}
I20260619 14:51:19.367178 22059 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32c2516788e24d319e6ad1a6f0acbeb5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "32c2516788e24d319e6ad1a6f0acbeb5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32c2516788e24d319e6ad1a6f0acbeb5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42477 } } }
I20260619 14:51:19.367267 22059 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32c2516788e24d319e6ad1a6f0acbeb5 [sys.catalog]: This master's current role is: LEADER
I20260619 14:51:19.371111 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260619 14:51:19.371323 22073 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 32c2516788e24d319e6ad1a6f0acbeb5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:51:19.371405 22073 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:51:19.371503 22067 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:51:19.372500 22067 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:51:19.380223 22067 catalog_manager.cc:1347] Generated new cluster ID: 68e0904cfe214e39be7184000c893fd0
I20260619 14:51:19.380316 22067 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:51:19.384121 22056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:19.373726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47891 (local address 127.0.98.62:42477)
0619 14:51:19.373842 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:19.373847 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:19.374889 (+ 1042us) server_negotiation.cc:407] Connection header received
0619 14:51:19.375015 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:19.375019 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:19.375076 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:19.375197 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:19.376095 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:19.377006 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:19.378167 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:19.378489 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:19.378591 (+ 102us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:19.380595 (+ 2004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:19.380599 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:19.380601 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:19.380627 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:19.382248 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:19.382774 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:19.382778 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:19.382779 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:19.382864 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:19.383211 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:19.383214 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:19.383216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:19.383405 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:19.383575 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:19.383849 (+ 274us) server_negotiation.cc:299] Negotiation successful
0619 14:51:19.383953 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260619 14:51:19.397876 22067 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:51:19.399075 22067 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:51:19.404671 22067 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 32c2516788e24d319e6ad1a6f0acbeb5: Generated new TSK 0
I20260619 14:51:19.405462 22067 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:51:19.415473 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 21985
W20260619 14:51:19.421545 21970 connection.cc:535] client connection to 127.0.98.62:42477 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:42477): Connection reset by peer (error 104)
2026-06-19T14:51:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1532 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781880383132139-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:51:19 dist-test-slave-2sb1 krb5kdc[22084](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:51:19 dist-test-slave-2sb1 krb5kdc[22084](info): set up 2 sockets
Jun 19 14:51:19 dist-test-slave-2sb1 krb5kdc[22084](info): commencing operation
krb5kdc: starting...
W20260619 14:51:20.553296 392 mini_kdc.cc:121] Time spent starting KDC: real 1.116s user 0.007s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781880383132139-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781880383132139-392-0/krb5kdc/test-user.keytab.
Jun 19 14:51:20 dist-test-slave-2sb1 krb5kdc[22084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:51:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:51:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 14:51:20.700912 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781880383132139-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781880383132139-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781880383132139-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38483
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38483
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:51:20.816919 22097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:51:20.817319 22097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:51:20.817369 22097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:51:20.821698 22097 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:51:20.821786 22097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:51:20.821807 22097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:51:20.821826 22097 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:51:20.821841 22097 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:51:20.827834 22097 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781880383132139-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781880383132139-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38483
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38483
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22097
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781880383132139-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:51:20.829268 22097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:51:20.830363 22097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 14:51:20.839998 22097 server_base.cc:1034] running on GCE node
W20260619 14:51:20.839949 22105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:51:20.839949 22103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:51:20.839942 22102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:51:20.841010 22097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:51:20.842489 22097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:51:20.843737 22097 hybrid_clock.cc:648] HybridClock initialized: now 1781880680843704 us; error 51 us; skew 500 ppm
Jun 19 14:51:20 dist-test-slave-2sb1 krb5kdc[22084](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880680, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:51:20.847317 22097 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 14:51:20.848507 22097 webserver.cc:460] Webserver started at http://127.0.98.62:45065/ using document root <none> and password file <none>
I20260619 14:51:20.849272 22097 fs_manager.cc:359] Metadata directory not provided
I20260619 14:51:20.849376 22097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:51:20.849632 22097 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:51:20.851943 22097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781880383132139-392-0/minicluster-data/master-0/data/instance:
uuid: "377ccd876029477aaedaef49bc005189"
format_stamp: "Formatted at 2026-06-19 14:51:20 on dist-test-slave-2sb1"
server_key: "0395e9b29e07f31a5df532f2b460a8de"
server_key_iv: "296fb8406a05e3a4d3ac30c43060955e"
server_key_version: "encryptionkey@0"
I20260619 14:51:20.852564 22097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781880383132139-392-0/minicluster-data/master-0/wal/instance:
uuid: "377ccd876029477aaedaef49bc005189"
format_stamp: "Formatted at 2026-06-19 14:51:20 on dist-test-slave-2sb1"
server_key: "0395e9b29e07f31a5df532f2b460a8de"
server_key_iv: "296fb8406a05e3a4d3ac30c43060955e"
server_key_version: "encryptionkey@0"
I20260619 14:51:20.856875 22097 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260619 14:51:20.859771 22112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:20.861270 22097 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:51:20.861439 22097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781880383132139-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781880383132139-392-0/minicluster-data/master-0/wal
uuid: "377ccd876029477aaedaef49bc005189"
format_stamp: "Formatted at 2026-06-19 14:51:20 on dist-test-slave-2sb1"
server_key: "0395e9b29e07f31a5df532f2b460a8de"
server_key_iv: "296fb8406a05e3a4d3ac30c43060955e"
server_key_version: "encryptionkey@0"
I20260619 14:51:20.861572 22097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781880383132139-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781880383132139-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781880383132139-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:51:20.882359 22097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:51:20.886039 22097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:51:20.886281 22097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:51:20.894025 22097 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38483
I20260619 14:51:20.894130 22164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38483 every 8 connection(s)
I20260619 14:51:20.895115 22097 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:51:20.898430 22165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:51:20.898888 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 22097
I20260619 14:51:20.899034 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781880383132139-392-0/minicluster-data/master-0/wal/instance
I20260619 14:51:20.899345 392 external_mini_cluster.cc:1428] Setting key 29bfc398b42dd93077df18d89e4a82f4
I20260619 14:51:20.907405 22165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 377ccd876029477aaedaef49bc005189: Bootstrap starting.
I20260619 14:51:20.910002 22165 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 377ccd876029477aaedaef49bc005189: Neither blocks nor log segments found. Creating new log.
I20260619 14:51:20.910279 22168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:20.901008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40533 (local address 127.0.98.62:38483)
0619 14:51:20.901510 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:20.901540 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:20.901579 (+ 39us) server_negotiation.cc:407] Connection header received
0619 14:51:20.902372 (+ 793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:20.902395 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:20.902817 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:20.903135 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:20.903663 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:20.904878 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:20.906198 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:20.906774 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:20.906920 (+ 146us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:20.908255 (+ 1335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:20.908260 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:20.908275 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:51:20.908795 (+ 520us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:20.908866 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:51:20.908893 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:20.909065 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:20.909290 (+ 225us) server_negotiation.cc:299] Negotiation successful
0619 14:51:20.909445 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":131,"threads_started":1}
I20260619 14:51:20.911001 22165 log.cc:824] T 00000000000000000000000000000000 P 377ccd876029477aaedaef49bc005189: Log is configured to *not* fsync() on all Append() calls
I20260619 14:51:20.913543 22165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 377ccd876029477aaedaef49bc005189: No bootstrap required, opened a new log
I20260619 14:51:20.913806 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:51:20.918610 22165 raft_consensus.cc:348] T 00000000000000000000000000000000 P 377ccd876029477aaedaef49bc005189 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "377ccd876029477aaedaef49bc005189" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38483 } }
I20260619 14:51:20.918892 22165 raft_consensus.cc:374] T 00000000000000000000000000000000 P 377ccd876029477aaedaef49bc005189 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:51:20.918936 22165 raft_consensus.cc:729] T 00000000000000000000000000000000 P 377ccd876029477aaedaef49bc005189 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 377ccd876029477aaedaef49bc005189, State: Initialized, Role: FOLLOWER
I20260619 14:51:20.919462 22165 consensus_queue.cc:260] T 00000000000000000000000000000000 P 377ccd876029477aaedaef49bc005189 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "377ccd876029477aaedaef49bc005189" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38483 } }
I20260619 14:51:20.919621 22165 raft_consensus.cc:388] T 00000000000000000000000000000000 P 377ccd876029477aaedaef49bc005189 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:51:20.919693 22165 raft_consensus.cc:482] T 00000000000000000000000000000000 P 377ccd876029477aaedaef49bc005189 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:51:20.919821 22165 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 377ccd876029477aaedaef49bc005189 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:51:20.920925 22165 raft_consensus.cc:504] T 00000000000000000000000000000000 P 377ccd876029477aaedaef49bc005189 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "377ccd876029477aaedaef49bc005189" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38483 } }
I20260619 14:51:20.921059 22168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:20.916240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36177 (local address 127.0.98.62:38483)
0619 14:51:20.916402 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:20.916407 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:20.916470 (+ 63us) server_negotiation.cc:407] Connection header received
0619 14:51:20.916597 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:20.916600 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:20.916674 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:20.916762 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:20.917116 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:20.917931 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:20.919141 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:20.919405 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:20.919529 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:20.920500 (+ 971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:20.920504 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:20.920506 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:51:20.920520 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:20.920555 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:51:20.920566 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:20.920685 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:20.920833 (+ 148us) server_negotiation.cc:299] Negotiation successful
0619 14:51:20.920872 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260619 14:51:20.921383 22165 leader_election.cc:302] T 00000000000000000000000000000000 P 377ccd876029477aaedaef49bc005189 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 377ccd876029477aaedaef49bc005189; no voters:
I20260619 14:51:20.921756 22165 leader_election.cc:288] T 00000000000000000000000000000000 P 377ccd876029477aaedaef49bc005189 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:51:20.921833 22175 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 377ccd876029477aaedaef49bc005189 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:51:20.922081 22175 raft_consensus.cc:686] T 00000000000000000000000000000000 P 377ccd876029477aaedaef49bc005189 [term 1 LEADER]: Becoming Leader. State: Replica: 377ccd876029477aaedaef49bc005189, State: Running, Role: LEADER
I20260619 14:51:20.922401 22175 consensus_queue.cc:237] T 00000000000000000000000000000000 P 377ccd876029477aaedaef49bc005189 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "377ccd876029477aaedaef49bc005189" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38483 } }
I20260619 14:51:20.923213 22165 sys_catalog.cc:564] T 00000000000000000000000000000000 P 377ccd876029477aaedaef49bc005189 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:51:20.923650 22175 sys_catalog.cc:455] T 00000000000000000000000000000000 P 377ccd876029477aaedaef49bc005189 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 377ccd876029477aaedaef49bc005189. Latest consensus state: current_term: 1 leader_uuid: "377ccd876029477aaedaef49bc005189" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "377ccd876029477aaedaef49bc005189" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38483 } } }
I20260619 14:51:20.923768 22175 sys_catalog.cc:458] T 00000000000000000000000000000000 P 377ccd876029477aaedaef49bc005189 [sys.catalog]: This master's current role is: LEADER
I20260619 14:51:20.923952 22175 sys_catalog.cc:455] T 00000000000000000000000000000000 P 377ccd876029477aaedaef49bc005189 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "377ccd876029477aaedaef49bc005189" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "377ccd876029477aaedaef49bc005189" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38483 } } }
I20260619 14:51:20.924043 22175 sys_catalog.cc:458] T 00000000000000000000000000000000 P 377ccd876029477aaedaef49bc005189 [sys.catalog]: This master's current role is: LEADER
I20260619 14:51:20.924218 22179 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:51:20.927698 22179 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:51:20.934517 22179 catalog_manager.cc:1347] Generated new cluster ID: ddb1d26052554f908c1282799d4e49ea
I20260619 14:51:20.934618 22179 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:51:20.941251 22179 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:51:20.942456 22179 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:51:20.949613 22179 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 377ccd876029477aaedaef49bc005189: Generated new TSK 0
I20260619 14:51:20.950469 22179 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:51:20.959510 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 22097
W20260619 14:51:20.965162 22082 connection.cc:535] client connection to 127.0.98.62:38483 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:38483): Connection reset by peer (error 104)
2026-06-19T14:51:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1543 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781880383132139-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:51:21 dist-test-slave-2sb1 krb5kdc[22196](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:51:21 dist-test-slave-2sb1 krb5kdc[22196](info): set up 2 sockets
Jun 19 14:51:21 dist-test-slave-2sb1 krb5kdc[22196](info): commencing operation
krb5kdc: starting...
W20260619 14:51:22.061383 392 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781880383132139-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781880383132139-392-0/krb5kdc/test-user.keytab.
Jun 19 14:51:22 dist-test-slave-2sb1 krb5kdc[22196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880682, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:51:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:51:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 14:51:22.214022 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781880383132139-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781880383132139-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781880383132139-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45325
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45325
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:51:22.341424 22209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:51:22.341743 22209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:51:22.341787 22209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:51:22.346292 22209 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:51:22.346383 22209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:51:22.346405 22209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:51:22.346426 22209 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:51:22.346443 22209 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:51:22.352501 22209 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781880383132139-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781880383132139-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45325
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22209
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781880383132139-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:51:22.353739 22209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:51:22.354841 22209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:51:22.365618 22214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:51:22.365618 22215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:51:22.365659 22217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:51:22.367154 22209 server_base.cc:1034] running on GCE node
I20260619 14:51:22.367861 22209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:51:22.369552 22209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:51:22.371299 22209 hybrid_clock.cc:648] HybridClock initialized: now 1781880682371230 us; error 85 us; skew 500 ppm
Jun 19 14:51:22 dist-test-slave-2sb1 krb5kdc[22196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880682, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:51:22.381577 22209 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 14:51:22.382848 22209 webserver.cc:460] Webserver started at http://127.0.98.62:33547/ using document root <none> and password file <none>
I20260619 14:51:22.383514 22209 fs_manager.cc:359] Metadata directory not provided
I20260619 14:51:22.383585 22209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:51:22.383823 22209 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:51:22.386407 22209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781880383132139-392-0/minicluster-data/master-0/data/instance:
uuid: "2c900e18aa5540b8a3736e04b3eaa4cd"
format_stamp: "Formatted at 2026-06-19 14:51:22 on dist-test-slave-2sb1"
server_key: "2701958a3092f47763c67b14f1fee85b"
server_key_iv: "ffdb9b7a56d0a9bcdf2bc8a52a222a50"
server_key_version: "encryptionkey@0"
I20260619 14:51:22.387249 22209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781880383132139-392-0/minicluster-data/master-0/wal/instance:
uuid: "2c900e18aa5540b8a3736e04b3eaa4cd"
format_stamp: "Formatted at 2026-06-19 14:51:22 on dist-test-slave-2sb1"
server_key: "2701958a3092f47763c67b14f1fee85b"
server_key_iv: "ffdb9b7a56d0a9bcdf2bc8a52a222a50"
server_key_version: "encryptionkey@0"
I20260619 14:51:22.392171 22209 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 14:51:22.395709 22224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:22.397660 22209 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 14:51:22.397889 22209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781880383132139-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781880383132139-392-0/minicluster-data/master-0/wal
uuid: "2c900e18aa5540b8a3736e04b3eaa4cd"
format_stamp: "Formatted at 2026-06-19 14:51:22 on dist-test-slave-2sb1"
server_key: "2701958a3092f47763c67b14f1fee85b"
server_key_iv: "ffdb9b7a56d0a9bcdf2bc8a52a222a50"
server_key_version: "encryptionkey@0"
I20260619 14:51:22.398061 22209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781880383132139-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781880383132139-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781880383132139-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:51:22.404002 22209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:51:22.408250 22209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:51:22.408620 22209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:51:22.417237 22209 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45325
I20260619 14:51:22.417264 22276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45325 every 8 connection(s)
I20260619 14:51:22.418488 22209 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:51:22.422415 22277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:51:22.427335 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 22209
I20260619 14:51:22.427510 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781880383132139-392-0/minicluster-data/master-0/wal/instance
I20260619 14:51:22.427860 392 external_mini_cluster.cc:1428] Setting key 0d2bbfa01ab8de5d49ec513edbd4c271
I20260619 14:51:22.429947 22277 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c900e18aa5540b8a3736e04b3eaa4cd: Bootstrap starting.
I20260619 14:51:22.432766 22277 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c900e18aa5540b8a3736e04b3eaa4cd: Neither blocks nor log segments found. Creating new log.
I20260619 14:51:22.433647 22277 log.cc:824] T 00000000000000000000000000000000 P 2c900e18aa5540b8a3736e04b3eaa4cd: Log is configured to *not* fsync() on all Append() calls
I20260619 14:51:22.434346 22280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:22.429587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49297 (local address 127.0.98.62:45325)
0619 14:51:22.430119 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:22.430150 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:22.430196 (+ 46us) server_negotiation.cc:407] Connection header received
0619 14:51:22.431332 (+ 1136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:22.431361 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:22.431755 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:22.432154 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:22.433049 (+ 895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:22.433054 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:22.433078 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:51:22.433089 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:22.433144 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:51:22.433166 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:22.433297 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:22.433421 (+ 124us) server_negotiation.cc:299] Negotiation successful
0619 14:51:22.433609 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":137,"threads_started":1}
I20260619 14:51:22.435858 22277 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c900e18aa5540b8a3736e04b3eaa4cd: No bootstrap required, opened a new log
I20260619 14:51:22.437547 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:51:22.440459 22277 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c900e18aa5540b8a3736e04b3eaa4cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c900e18aa5540b8a3736e04b3eaa4cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45325 } }
I20260619 14:51:22.440832 22277 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2c900e18aa5540b8a3736e04b3eaa4cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:51:22.440917 22277 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c900e18aa5540b8a3736e04b3eaa4cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c900e18aa5540b8a3736e04b3eaa4cd, State: Initialized, Role: FOLLOWER
I20260619 14:51:22.441416 22277 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c900e18aa5540b8a3736e04b3eaa4cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c900e18aa5540b8a3736e04b3eaa4cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45325 } }
I20260619 14:51:22.441591 22277 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c900e18aa5540b8a3736e04b3eaa4cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:51:22.441660 22277 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c900e18aa5540b8a3736e04b3eaa4cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:51:22.441763 22277 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c900e18aa5540b8a3736e04b3eaa4cd [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:51:22.442020 22280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:22.440036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48689 (local address 127.0.98.62:45325)
0619 14:51:22.440176 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:22.440182 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:22.440408 (+ 226us) server_negotiation.cc:407] Connection header received
0619 14:51:22.440463 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:22.440465 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:22.440503 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:22.440595 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:22.441588 (+ 993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:22.441592 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:22.441595 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:51:22.441608 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:22.441640 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:51:22.441655 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:22.441745 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:22.441846 (+ 101us) server_negotiation.cc:299] Negotiation successful
0619 14:51:22.441888 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 14:51:22.443295 22277 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c900e18aa5540b8a3736e04b3eaa4cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c900e18aa5540b8a3736e04b3eaa4cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45325 } }
I20260619 14:51:22.443890 22277 leader_election.cc:302] T 00000000000000000000000000000000 P 2c900e18aa5540b8a3736e04b3eaa4cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c900e18aa5540b8a3736e04b3eaa4cd; no voters:
I20260619 14:51:22.444372 22277 leader_election.cc:288] T 00000000000000000000000000000000 P 2c900e18aa5540b8a3736e04b3eaa4cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:51:22.444638 22287 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c900e18aa5540b8a3736e04b3eaa4cd [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:51:22.444913 22287 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c900e18aa5540b8a3736e04b3eaa4cd [term 1 LEADER]: Becoming Leader. State: Replica: 2c900e18aa5540b8a3736e04b3eaa4cd, State: Running, Role: LEADER
I20260619 14:51:22.445261 22287 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c900e18aa5540b8a3736e04b3eaa4cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c900e18aa5540b8a3736e04b3eaa4cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45325 } }
I20260619 14:51:22.445945 22277 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c900e18aa5540b8a3736e04b3eaa4cd [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:51:22.446593 22287 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c900e18aa5540b8a3736e04b3eaa4cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c900e18aa5540b8a3736e04b3eaa4cd. Latest consensus state: current_term: 1 leader_uuid: "2c900e18aa5540b8a3736e04b3eaa4cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c900e18aa5540b8a3736e04b3eaa4cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45325 } } }
I20260619 14:51:22.446722 22287 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c900e18aa5540b8a3736e04b3eaa4cd [sys.catalog]: This master's current role is: LEADER
I20260619 14:51:22.447098 22293 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:51:22.447254 22288 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c900e18aa5540b8a3736e04b3eaa4cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2c900e18aa5540b8a3736e04b3eaa4cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c900e18aa5540b8a3736e04b3eaa4cd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45325 } } }
I20260619 14:51:22.447460 22288 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c900e18aa5540b8a3736e04b3eaa4cd [sys.catalog]: This master's current role is: LEADER
I20260619 14:51:22.450361 22293 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:51:22.456677 22293 catalog_manager.cc:1347] Generated new cluster ID: b17b1fde0d3940828dd50098955a1574
I20260619 14:51:22.456768 22293 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:51:22.473664 22293 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:51:22.474083 22293 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:51:22.485168 22293 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2c900e18aa5540b8a3736e04b3eaa4cd: Generated new TSK 0
I20260619 14:51:22.485965 22293 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:51:22.495048 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 22209
2026-06-19T14:51:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1535 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781880383132139-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:51:22 dist-test-slave-2sb1 krb5kdc[22308](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:51:22 dist-test-slave-2sb1 krb5kdc[22308](info): set up 2 sockets
Jun 19 14:51:22 dist-test-slave-2sb1 krb5kdc[22308](info): commencing operation
krb5kdc: starting...
W20260619 14:51:23.573304 392 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781880383132139-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781880383132139-392-0/krb5kdc/test-user.keytab.
Jun 19 14:51:23 dist-test-slave-2sb1 krb5kdc[22308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880683, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:51:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:51:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 14:51:23.758502 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781880383132139-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781880383132139-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781880383132139-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40621
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40621
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:51:23.904579 22321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:51:23.904935 22321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:51:23.905200 22321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:51:23.909850 22321 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:51:23.909941 22321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:51:23.909962 22321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:51:23.909979 22321 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:51:23.909994 22321 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:51:23.915848 22321 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781880383132139-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781880383132139-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40621
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40621
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22321
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781880383132139-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:51:23.917071 22321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:51:23.918165 22321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260619 14:51:23.928247 22321 server_base.cc:1034] running on GCE node
W20260619 14:51:23.928232 22327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:51:23.928232 22329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:51:23.928493 22326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:51:23.929236 22321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:51:23.930593 22321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:51:23.931829 22321 hybrid_clock.cc:648] HybridClock initialized: now 1781880683931744 us; error 113 us; skew 500 ppm
Jun 19 14:51:23 dist-test-slave-2sb1 krb5kdc[22308](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880683, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:51:23.935674 22321 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 14:51:23.936846 22321 webserver.cc:460] Webserver started at http://127.0.98.62:40175/ using document root <none> and password file <none>
I20260619 14:51:23.937444 22321 fs_manager.cc:359] Metadata directory not provided
I20260619 14:51:23.937522 22321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:51:23.937719 22321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:51:23.939910 22321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781880383132139-392-0/minicluster-data/master-0/data/instance:
uuid: "65c628b7cee547afb76481cd6a9eef67"
format_stamp: "Formatted at 2026-06-19 14:51:23 on dist-test-slave-2sb1"
server_key: "d7dd22f40fa2e679a2df1bce46bb141c"
server_key_iv: "c256c2b395988b7e8405e0fb3f90eae1"
server_key_version: "encryptionkey@0"
I20260619 14:51:23.940469 22321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781880383132139-392-0/minicluster-data/master-0/wal/instance:
uuid: "65c628b7cee547afb76481cd6a9eef67"
format_stamp: "Formatted at 2026-06-19 14:51:23 on dist-test-slave-2sb1"
server_key: "d7dd22f40fa2e679a2df1bce46bb141c"
server_key_iv: "c256c2b395988b7e8405e0fb3f90eae1"
server_key_version: "encryptionkey@0"
I20260619 14:51:23.945518 22321 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.003s
I20260619 14:51:23.948457 22336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:23.950204 22321 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 14:51:23.950335 22321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781880383132139-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781880383132139-392-0/minicluster-data/master-0/wal
uuid: "65c628b7cee547afb76481cd6a9eef67"
format_stamp: "Formatted at 2026-06-19 14:51:23 on dist-test-slave-2sb1"
server_key: "d7dd22f40fa2e679a2df1bce46bb141c"
server_key_iv: "c256c2b395988b7e8405e0fb3f90eae1"
server_key_version: "encryptionkey@0"
I20260619 14:51:23.950454 22321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781880383132139-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781880383132139-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781880383132139-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:51:23.967927 22321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:51:23.972076 22321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:51:23.972259 22321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:51:23.980280 22321 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40621
I20260619 14:51:23.980296 22388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40621 every 8 connection(s)
I20260619 14:51:23.981335 22321 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:51:23.984628 22389 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:51:23.987857 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 22321
I20260619 14:51:23.988001 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781880383132139-392-0/minicluster-data/master-0/wal/instance
I20260619 14:51:23.988322 392 external_mini_cluster.cc:1428] Setting key fdf708de2588cc5388f531e46c913e36
I20260619 14:51:23.992545 22389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65c628b7cee547afb76481cd6a9eef67: Bootstrap starting.
I20260619 14:51:23.995684 22389 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65c628b7cee547afb76481cd6a9eef67: Neither blocks nor log segments found. Creating new log.
I20260619 14:51:23.996713 22389 log.cc:824] T 00000000000000000000000000000000 P 65c628b7cee547afb76481cd6a9eef67: Log is configured to *not* fsync() on all Append() calls
Jun 19 14:51:23 dist-test-slave-2sb1 krb5kdc[22308](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880683, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:51:24.000089 22389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65c628b7cee547afb76481cd6a9eef67: No bootstrap required, opened a new log
I20260619 14:51:24.004885 22389 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65c628b7cee547afb76481cd6a9eef67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65c628b7cee547afb76481cd6a9eef67" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40621 } }
I20260619 14:51:24.005216 22389 raft_consensus.cc:374] T 00000000000000000000000000000000 P 65c628b7cee547afb76481cd6a9eef67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:51:24.005278 22389 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65c628b7cee547afb76481cd6a9eef67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65c628b7cee547afb76481cd6a9eef67, State: Initialized, Role: FOLLOWER
I20260619 14:51:24.006398 22389 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65c628b7cee547afb76481cd6a9eef67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65c628b7cee547afb76481cd6a9eef67" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40621 } }
I20260619 14:51:24.006628 22389 raft_consensus.cc:388] T 00000000000000000000000000000000 P 65c628b7cee547afb76481cd6a9eef67 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:51:24.006717 22389 raft_consensus.cc:482] T 00000000000000000000000000000000 P 65c628b7cee547afb76481cd6a9eef67 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:51:24.006847 22389 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65c628b7cee547afb76481cd6a9eef67 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:51:24.006987 22392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:23.989864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60947 (local address 127.0.98.62:40621)
0619 14:51:23.990421 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:23.990463 (+ 42us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:23.990505 (+ 42us) server_negotiation.cc:407] Connection header received
0619 14:51:23.991375 (+ 870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:23.991397 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:23.991813 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:23.992145 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:23.993350 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:23.994453 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:23.995907 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:23.996480 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:23.996615 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:24.000373 (+ 3758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:24.000379 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:24.000397 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:24.000438 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:24.002677 (+ 2239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:24.003281 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:24.003287 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:24.003292 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:24.003429 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:24.004108 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:24.004113 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:24.004115 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:24.004632 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:24.004984 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:24.005223 (+ 239us) server_negotiation.cc:299] Negotiation successful
0619 14:51:24.006007 (+ 784us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":175,"threads_started":1}
I20260619 14:51:24.008082 22389 raft_consensus.cc:504] T 00000000000000000000000000000000 P 65c628b7cee547afb76481cd6a9eef67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65c628b7cee547afb76481cd6a9eef67" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40621 } }
I20260619 14:51:24.008489 22389 leader_election.cc:302] T 00000000000000000000000000000000 P 65c628b7cee547afb76481cd6a9eef67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 65c628b7cee547afb76481cd6a9eef67; no voters:
I20260619 14:51:24.008908 22389 leader_election.cc:288] T 00000000000000000000000000000000 P 65c628b7cee547afb76481cd6a9eef67 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:51:24.008997 22394 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 65c628b7cee547afb76481cd6a9eef67 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:51:24.009204 22394 raft_consensus.cc:686] T 00000000000000000000000000000000 P 65c628b7cee547afb76481cd6a9eef67 [term 1 LEADER]: Becoming Leader. State: Replica: 65c628b7cee547afb76481cd6a9eef67, State: Running, Role: LEADER
I20260619 14:51:24.009656 22394 consensus_queue.cc:237] T 00000000000000000000000000000000 P 65c628b7cee547afb76481cd6a9eef67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65c628b7cee547afb76481cd6a9eef67" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40621 } }
I20260619 14:51:24.010418 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:51:24.010608 22389 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65c628b7cee547afb76481cd6a9eef67 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:51:24.011724 22396 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65c628b7cee547afb76481cd6a9eef67 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65c628b7cee547afb76481cd6a9eef67. Latest consensus state: current_term: 1 leader_uuid: "65c628b7cee547afb76481cd6a9eef67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65c628b7cee547afb76481cd6a9eef67" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40621 } } }
I20260619 14:51:24.012343 22396 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65c628b7cee547afb76481cd6a9eef67 [sys.catalog]: This master's current role is: LEADER
I20260619 14:51:24.013089 22395 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65c628b7cee547afb76481cd6a9eef67 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "65c628b7cee547afb76481cd6a9eef67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65c628b7cee547afb76481cd6a9eef67" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40621 } } }
I20260619 14:51:24.013231 22395 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65c628b7cee547afb76481cd6a9eef67 [sys.catalog]: This master's current role is: LEADER
I20260619 14:51:24.014590 22407 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:51:24.018267 22407 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:51:24.025017 22392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:24.013395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43145 (local address 127.0.98.62:40621)
0619 14:51:24.014041 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:24.014044 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:24.014059 (+ 15us) server_negotiation.cc:407] Connection header received
0619 14:51:24.014097 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:24.014100 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:24.014137 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:24.014213 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:24.015214 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:24.015800 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:24.017029 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:24.017384 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:24.017516 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:24.019949 (+ 2433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:24.019954 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:24.019957 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:24.019993 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:24.021979 (+ 1986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:24.022706 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:24.022710 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:24.022712 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:24.022822 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:24.023179 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:24.023182 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:24.023184 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:24.023425 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:24.023582 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:24.024581 (+ 999us) server_negotiation.cc:299] Negotiation successful
0619 14:51:24.024836 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581}
I20260619 14:51:24.025961 22407 catalog_manager.cc:1347] Generated new cluster ID: cce0cfffaf8f4abbafc248dee2a63bd3
I20260619 14:51:24.026105 22407 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:51:24.074878 22407 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:51:24.076148 22407 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:51:24.090066 22407 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 65c628b7cee547afb76481cd6a9eef67: Generated new TSK 0
I20260619 14:51:24.090874 22407 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:51:24.104825 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 22321
2026-06-19T14:51:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1611 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781880383132139-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:51:24 dist-test-slave-2sb1 krb5kdc[22420](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:51:24 dist-test-slave-2sb1 krb5kdc[22420](info): set up 2 sockets
Jun 19 14:51:24 dist-test-slave-2sb1 krb5kdc[22420](info): commencing operation
krb5kdc: starting...
W20260619 14:51:25.233639 392 mini_kdc.cc:121] Time spent starting KDC: real 1.107s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781880383132139-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781880383132139-392-0/krb5kdc/test-user.keytab.
Jun 19 14:51:25 dist-test-slave-2sb1 krb5kdc[22420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880685, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:51:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:51:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 14:51:25.385006 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781880383132139-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781880383132139-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781880383132139-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42609
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42609
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:51:25.514027 22433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:51:25.514379 22433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:51:25.514456 22433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:51:25.518927 22433 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:51:25.519061 22433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:51:25.519096 22433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:51:25.519140 22433 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:51:25.519194 22433 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:51:25.524796 22433 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781880383132139-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781880383132139-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42609
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42609
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22433
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781880383132139-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:51:25.525993 22433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:51:25.527025 22433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:51:25.536530 22441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:51:25.536592 22433 server_base.cc:1034] running on GCE node
W20260619 14:51:25.536610 22439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:51:25.536530 22438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:51:25.537750 22433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:51:25.539199 22433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:51:25.540493 22433 hybrid_clock.cc:648] HybridClock initialized: now 1781880685540460 us; error 51 us; skew 500 ppm
Jun 19 14:51:25 dist-test-slave-2sb1 krb5kdc[22420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880685, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:51:25.544694 22433 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 14:51:25.545923 22433 webserver.cc:460] Webserver started at http://127.0.98.62:42105/ using document root <none> and password file <none>
I20260619 14:51:25.546593 22433 fs_manager.cc:359] Metadata directory not provided
I20260619 14:51:25.546705 22433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:51:25.546973 22433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:51:25.549391 22433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781880383132139-392-0/minicluster-data/master-0/data/instance:
uuid: "ecbce27f3c80489ead5132db015a33d7"
format_stamp: "Formatted at 2026-06-19 14:51:25 on dist-test-slave-2sb1"
server_key: "79752ae27c1834e5fc5bbfb2fd186cc0"
server_key_iv: "c6e1924f88a2febe017d8e58d4f9b603"
server_key_version: "encryptionkey@0"
I20260619 14:51:25.550055 22433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781880383132139-392-0/minicluster-data/master-0/wal/instance:
uuid: "ecbce27f3c80489ead5132db015a33d7"
format_stamp: "Formatted at 2026-06-19 14:51:25 on dist-test-slave-2sb1"
server_key: "79752ae27c1834e5fc5bbfb2fd186cc0"
server_key_iv: "c6e1924f88a2febe017d8e58d4f9b603"
server_key_version: "encryptionkey@0"
I20260619 14:51:25.554600 22433 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260619 14:51:25.557561 22448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:25.559194 22433 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:51:25.559405 22433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781880383132139-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781880383132139-392-0/minicluster-data/master-0/wal
uuid: "ecbce27f3c80489ead5132db015a33d7"
format_stamp: "Formatted at 2026-06-19 14:51:25 on dist-test-slave-2sb1"
server_key: "79752ae27c1834e5fc5bbfb2fd186cc0"
server_key_iv: "c6e1924f88a2febe017d8e58d4f9b603"
server_key_version: "encryptionkey@0"
I20260619 14:51:25.559557 22433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781880383132139-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781880383132139-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781880383132139-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:51:25.587278 22433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:51:25.591332 22433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:51:25.591598 22433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:51:25.599589 22433 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42609
I20260619 14:51:25.599623 22500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42609 every 8 connection(s)
I20260619 14:51:25.600682 22433 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:51:25.604089 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 22433
I20260619 14:51:25.604259 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781880383132139-392-0/minicluster-data/master-0/wal/instance
I20260619 14:51:25.604517 22501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:51:25.604590 392 external_mini_cluster.cc:1428] Setting key 535f00c856321ecfd6719598d73246ea
I20260619 14:51:25.612326 22501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecbce27f3c80489ead5132db015a33d7: Bootstrap starting.
I20260619 14:51:25.615440 22501 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ecbce27f3c80489ead5132db015a33d7: Neither blocks nor log segments found. Creating new log.
I20260619 14:51:25.616381 22501 log.cc:824] T 00000000000000000000000000000000 P ecbce27f3c80489ead5132db015a33d7: Log is configured to *not* fsync() on all Append() calls
I20260619 14:51:25.616600 22504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:25.606321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35117 (local address 127.0.98.62:42609)
0619 14:51:25.606832 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:25.606856 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:25.607017 (+ 161us) server_negotiation.cc:407] Connection header received
0619 14:51:25.608131 (+ 1114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:25.608160 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:25.608715 (+ 555us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:25.609135 (+ 420us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:25.609581 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:25.611253 (+ 1672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:25.612415 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:25.613143 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:25.613351 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:25.614313 (+ 962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:25.614320 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:25.614337 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:51:25.614876 (+ 539us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:25.614949 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:51:25.614982 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:25.615087 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:25.615298 (+ 211us) server_negotiation.cc:299] Negotiation successful
0619 14:51:25.615542 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":148,"threads_started":1}
I20260619 14:51:25.618963 22501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecbce27f3c80489ead5132db015a33d7: No bootstrap required, opened a new log
I20260619 14:51:25.620543 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:51:25.624262 22501 raft_consensus.cc:348] T 00000000000000000000000000000000 P ecbce27f3c80489ead5132db015a33d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecbce27f3c80489ead5132db015a33d7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42609 } }
I20260619 14:51:25.624639 22501 raft_consensus.cc:374] T 00000000000000000000000000000000 P ecbce27f3c80489ead5132db015a33d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:51:25.624728 22501 raft_consensus.cc:729] T 00000000000000000000000000000000 P ecbce27f3c80489ead5132db015a33d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecbce27f3c80489ead5132db015a33d7, State: Initialized, Role: FOLLOWER
I20260619 14:51:25.625263 22501 consensus_queue.cc:260] T 00000000000000000000000000000000 P ecbce27f3c80489ead5132db015a33d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecbce27f3c80489ead5132db015a33d7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42609 } }
I20260619 14:51:25.625478 22501 raft_consensus.cc:388] T 00000000000000000000000000000000 P ecbce27f3c80489ead5132db015a33d7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:51:25.625582 22501 raft_consensus.cc:482] T 00000000000000000000000000000000 P ecbce27f3c80489ead5132db015a33d7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:51:25.625706 22501 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ecbce27f3c80489ead5132db015a33d7 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:51:25.627099 22501 raft_consensus.cc:504] T 00000000000000000000000000000000 P ecbce27f3c80489ead5132db015a33d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecbce27f3c80489ead5132db015a33d7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42609 } }
I20260619 14:51:25.627596 22501 leader_election.cc:302] T 00000000000000000000000000000000 P ecbce27f3c80489ead5132db015a33d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ecbce27f3c80489ead5132db015a33d7; no voters:
I20260619 14:51:25.628064 22501 leader_election.cc:288] T 00000000000000000000000000000000 P ecbce27f3c80489ead5132db015a33d7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:51:25.628152 22511 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ecbce27f3c80489ead5132db015a33d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:51:25.628407 22504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:25.623133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58211 (local address 127.0.98.62:42609)
0619 14:51:25.623297 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:25.623302 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:25.623321 (+ 19us) server_negotiation.cc:407] Connection header received
0619 14:51:25.623466 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:25.623470 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:25.623525 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:25.623622 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:25.624060 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:25.624967 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:25.626088 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:25.626462 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:25.626588 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:25.627744 (+ 1156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:25.627748 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:25.627752 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:51:25.627769 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:25.627804 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:51:25.627819 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:25.627927 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:25.628167 (+ 240us) server_negotiation.cc:299] Negotiation successful
0619 14:51:25.628229 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260619 14:51:25.628446 22511 raft_consensus.cc:686] T 00000000000000000000000000000000 P ecbce27f3c80489ead5132db015a33d7 [term 1 LEADER]: Becoming Leader. State: Replica: ecbce27f3c80489ead5132db015a33d7, State: Running, Role: LEADER
I20260619 14:51:25.628829 22511 consensus_queue.cc:237] T 00000000000000000000000000000000 P ecbce27f3c80489ead5132db015a33d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecbce27f3c80489ead5132db015a33d7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42609 } }
I20260619 14:51:25.629693 22501 sys_catalog.cc:564] T 00000000000000000000000000000000 P ecbce27f3c80489ead5132db015a33d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:51:25.630167 22511 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecbce27f3c80489ead5132db015a33d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ecbce27f3c80489ead5132db015a33d7. Latest consensus state: current_term: 1 leader_uuid: "ecbce27f3c80489ead5132db015a33d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecbce27f3c80489ead5132db015a33d7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42609 } } }
I20260619 14:51:25.630299 22511 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecbce27f3c80489ead5132db015a33d7 [sys.catalog]: This master's current role is: LEADER
I20260619 14:51:25.630570 22511 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecbce27f3c80489ead5132db015a33d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ecbce27f3c80489ead5132db015a33d7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecbce27f3c80489ead5132db015a33d7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42609 } } }
I20260619 14:51:25.630724 22515 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:51:25.630744 22511 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecbce27f3c80489ead5132db015a33d7 [sys.catalog]: This master's current role is: LEADER
I20260619 14:51:25.634557 22515 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:51:25.642199 22515 catalog_manager.cc:1347] Generated new cluster ID: 459540832fe24061a0f144bc15de8643
I20260619 14:51:25.642302 22515 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:51:25.654212 22515 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:51:25.655653 22515 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:51:25.661810 22515 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ecbce27f3c80489ead5132db015a33d7: Generated new TSK 0
I20260619 14:51:25.662616 22515 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:51:25.666728 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 22433
2026-06-19T14:51:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1561 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781880383132139-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:51:25 dist-test-slave-2sb1 krb5kdc[22532](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:51:25 dist-test-slave-2sb1 krb5kdc[22532](info): set up 2 sockets
Jun 19 14:51:25 dist-test-slave-2sb1 krb5kdc[22532](info): commencing operation
krb5kdc: starting...
W20260619 14:51:26.791263 392 mini_kdc.cc:121] Time spent starting KDC: real 1.103s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781880383132139-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781880383132139-392-0/krb5kdc/test-user.keytab.
Jun 19 14:51:26 dist-test-slave-2sb1 krb5kdc[22532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880686, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:51:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:51:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 14:51:26.932935 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781880383132139-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781880383132139-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781880383132139-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34485
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34485
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:51:27.062412 22545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:51:27.062723 22545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:51:27.062768 22545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:51:27.067375 22545 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:51:27.067473 22545 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:51:27.067494 22545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:51:27.067512 22545 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:51:27.067528 22545 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:51:27.073478 22545 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781880383132139-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781880383132139-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34485
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34485
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22545
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781880383132139-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:51:27.074662 22545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:51:27.075740 22545 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:51:27.084743 22550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:51:27.084743 22551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:51:27.085042 22545 server_base.cc:1034] running on GCE node
W20260619 14:51:27.085103 22553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:51:27.085824 22545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:51:27.087186 22545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:51:27.088379 22545 hybrid_clock.cc:648] HybridClock initialized: now 1781880687088345 us; error 60 us; skew 500 ppm
Jun 19 14:51:27 dist-test-slave-2sb1 krb5kdc[22532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880687, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:51:27.091853 22545 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 14:51:27.093101 22545 webserver.cc:460] Webserver started at http://127.0.98.62:41807/ using document root <none> and password file <none>
I20260619 14:51:27.093703 22545 fs_manager.cc:359] Metadata directory not provided
I20260619 14:51:27.093816 22545 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:51:27.094065 22545 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:51:27.096351 22545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781880383132139-392-0/minicluster-data/master-0/data/instance:
uuid: "e195c8812f2e47ed880ee3ecc37173e6"
format_stamp: "Formatted at 2026-06-19 14:51:27 on dist-test-slave-2sb1"
server_key: "983cbd2551fd56a2fdb928df6b97040e"
server_key_iv: "b1daf3c514d405168444c6e81708dcbb"
server_key_version: "encryptionkey@0"
I20260619 14:51:27.097010 22545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781880383132139-392-0/minicluster-data/master-0/wal/instance:
uuid: "e195c8812f2e47ed880ee3ecc37173e6"
format_stamp: "Formatted at 2026-06-19 14:51:27 on dist-test-slave-2sb1"
server_key: "983cbd2551fd56a2fdb928df6b97040e"
server_key_iv: "b1daf3c514d405168444c6e81708dcbb"
server_key_version: "encryptionkey@0"
I20260619 14:51:27.101506 22545 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 14:51:27.104564 22560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:27.106074 22545 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:51:27.106266 22545 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781880383132139-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781880383132139-392-0/minicluster-data/master-0/wal
uuid: "e195c8812f2e47ed880ee3ecc37173e6"
format_stamp: "Formatted at 2026-06-19 14:51:27 on dist-test-slave-2sb1"
server_key: "983cbd2551fd56a2fdb928df6b97040e"
server_key_iv: "b1daf3c514d405168444c6e81708dcbb"
server_key_version: "encryptionkey@0"
I20260619 14:51:27.106485 22545 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781880383132139-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781880383132139-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781880383132139-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:51:27.111891 22545 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:51:27.115734 22545 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:51:27.115994 22545 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:51:27.125173 22612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34485 every 8 connection(s)
I20260619 14:51:27.125176 22545 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34485
I20260619 14:51:27.126267 22545 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:51:27.129591 22613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:51:27.130244 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 22545
I20260619 14:51:27.130424 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781880383132139-392-0/minicluster-data/master-0/wal/instance
I20260619 14:51:27.130730 392 external_mini_cluster.cc:1428] Setting key b216970f7bd77c88d79302f541bd2e24
I20260619 14:51:27.136889 22613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e195c8812f2e47ed880ee3ecc37173e6: Bootstrap starting.
I20260619 14:51:27.137055 22616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:27.132337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49325 (local address 127.0.98.62:34485)
0619 14:51:27.132845 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:27.132867 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:27.132912 (+ 45us) server_negotiation.cc:407] Connection header received
0619 14:51:27.133751 (+ 839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:27.133778 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:27.134155 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:27.134579 (+ 424us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:27.135503 (+ 924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:27.135512 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:27.135531 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:51:27.135544 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:27.135618 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:51:27.135645 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:27.135803 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:27.135905 (+ 102us) server_negotiation.cc:299] Negotiation successful
0619 14:51:27.136063 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":135,"threads_started":1}
I20260619 14:51:27.139571 22613 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e195c8812f2e47ed880ee3ecc37173e6: Neither blocks nor log segments found. Creating new log.
I20260619 14:51:27.140503 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:51:27.140501 22613 log.cc:824] T 00000000000000000000000000000000 P e195c8812f2e47ed880ee3ecc37173e6: Log is configured to *not* fsync() on all Append() calls
I20260619 14:51:27.143069 22613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e195c8812f2e47ed880ee3ecc37173e6: No bootstrap required, opened a new log
I20260619 14:51:27.145052 22616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:27.142896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59961 (local address 127.0.98.62:34485)
0619 14:51:27.143181 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:27.143186 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:27.143205 (+ 19us) server_negotiation.cc:407] Connection header received
0619 14:51:27.143398 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:27.143402 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:27.143439 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:27.143521 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:27.144478 (+ 957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:27.144481 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:27.144483 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0619 14:51:27.144495 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:27.144522 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0619 14:51:27.144532 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:27.144617 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:27.144745 (+ 128us) server_negotiation.cc:299] Negotiation successful
0619 14:51:27.144899 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169}
I20260619 14:51:27.147668 22613 raft_consensus.cc:348] T 00000000000000000000000000000000 P e195c8812f2e47ed880ee3ecc37173e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e195c8812f2e47ed880ee3ecc37173e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34485 } }
I20260619 14:51:27.147939 22613 raft_consensus.cc:374] T 00000000000000000000000000000000 P e195c8812f2e47ed880ee3ecc37173e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260619 14:51:27.147974 22613 raft_consensus.cc:729] T 00000000000000000000000000000000 P e195c8812f2e47ed880ee3ecc37173e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e195c8812f2e47ed880ee3ecc37173e6, State: Initialized, Role: FOLLOWER
I20260619 14:51:27.148442 22613 consensus_queue.cc:260] T 00000000000000000000000000000000 P e195c8812f2e47ed880ee3ecc37173e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e195c8812f2e47ed880ee3ecc37173e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34485 } }
I20260619 14:51:27.148579 22613 raft_consensus.cc:388] T 00000000000000000000000000000000 P e195c8812f2e47ed880ee3ecc37173e6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260619 14:51:27.148653 22613 raft_consensus.cc:482] T 00000000000000000000000000000000 P e195c8812f2e47ed880ee3ecc37173e6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260619 14:51:27.148792 22613 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e195c8812f2e47ed880ee3ecc37173e6 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:51:27.149844 22613 raft_consensus.cc:504] T 00000000000000000000000000000000 P e195c8812f2e47ed880ee3ecc37173e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e195c8812f2e47ed880ee3ecc37173e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34485 } }
I20260619 14:51:27.150225 22613 leader_election.cc:302] T 00000000000000000000000000000000 P e195c8812f2e47ed880ee3ecc37173e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e195c8812f2e47ed880ee3ecc37173e6; no voters:
I20260619 14:51:27.150617 22613 leader_election.cc:288] T 00000000000000000000000000000000 P e195c8812f2e47ed880ee3ecc37173e6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260619 14:51:27.150983 22623 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e195c8812f2e47ed880ee3ecc37173e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:51:27.151252 22623 raft_consensus.cc:686] T 00000000000000000000000000000000 P e195c8812f2e47ed880ee3ecc37173e6 [term 1 LEADER]: Becoming Leader. State: Replica: e195c8812f2e47ed880ee3ecc37173e6, State: Running, Role: LEADER
I20260619 14:51:27.151587 22623 consensus_queue.cc:237] T 00000000000000000000000000000000 P e195c8812f2e47ed880ee3ecc37173e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e195c8812f2e47ed880ee3ecc37173e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34485 } }
I20260619 14:51:27.151979 22613 sys_catalog.cc:564] T 00000000000000000000000000000000 P e195c8812f2e47ed880ee3ecc37173e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:51:27.153402 22624 sys_catalog.cc:455] T 00000000000000000000000000000000 P e195c8812f2e47ed880ee3ecc37173e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e195c8812f2e47ed880ee3ecc37173e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e195c8812f2e47ed880ee3ecc37173e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34485 } } }
I20260619 14:51:27.153532 22624 sys_catalog.cc:458] T 00000000000000000000000000000000 P e195c8812f2e47ed880ee3ecc37173e6 [sys.catalog]: This master's current role is: LEADER
I20260619 14:51:27.153911 22625 sys_catalog.cc:455] T 00000000000000000000000000000000 P e195c8812f2e47ed880ee3ecc37173e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e195c8812f2e47ed880ee3ecc37173e6. Latest consensus state: current_term: 1 leader_uuid: "e195c8812f2e47ed880ee3ecc37173e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e195c8812f2e47ed880ee3ecc37173e6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34485 } } }
I20260619 14:51:27.153971 22634 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:51:27.154014 22625 sys_catalog.cc:458] T 00000000000000000000000000000000 P e195c8812f2e47ed880ee3ecc37173e6 [sys.catalog]: This master's current role is: LEADER
I20260619 14:51:27.157374 22634 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:51:27.163712 22634 catalog_manager.cc:1347] Generated new cluster ID: 43dc4b7b455d4219802b8b88b49b6890
I20260619 14:51:27.163870 22634 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:51:27.195405 22634 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:51:27.195791 22634 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:51:27.204702 22634 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e195c8812f2e47ed880ee3ecc37173e6: Generated new TSK 0
I20260619 14:51:27.205515 22634 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260619 14:51:27.217465 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 22545
2026-06-19T14:51:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1550 ms)
[----------] 18 tests from AuthTokenIssuingTest (27950 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880383132139-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:51:27 dist-test-slave-2sb1 krb5kdc[22644](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:51:27 dist-test-slave-2sb1 krb5kdc[22644](info): set up 2 sockets
Jun 19 14:51:27 dist-test-slave-2sb1 krb5kdc[22644](info): commencing operation
krb5kdc: starting...
W20260619 14:51:28.329649 392 mini_kdc.cc:121] Time spent starting KDC: real 1.091s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880383132139-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880383132139-392-0/krb5kdc/test-user.keytab.
Jun 19 14:51:28 dist-test-slave-2sb1 krb5kdc[22644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880688, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:51:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:51:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 14:51:28.477386 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880383132139-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880383132139-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880383132139-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44253
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44253,127.0.98.61:42607,127.0.98.60:45231
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:51:28.597007 22657 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 14:51:28.597368 22657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:51:28.597397 22657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:51:28.597446 22657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:51:28.602363 22657 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:51:28.602458 22657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:51:28.602479 22657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:51:28.602499 22657 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:51:28.602516 22657 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:51:28.608754 22657 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880383132139-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880383132139-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44253,127.0.98.61:42607,127.0.98.60:45231
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22657
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880383132139-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:51:28.609977 22657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:51:28.611050 22657 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:51:28.621052 22662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:51:28.621197 22665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:51:28.621029 22663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:51:28.621392 22657 server_base.cc:1034] running on GCE node
I20260619 14:51:28.622211 22657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:51:28.623499 22657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:51:28.624761 22657 hybrid_clock.cc:648] HybridClock initialized: now 1781880688624647 us; error 133 us; skew 500 ppm
Jun 19 14:51:28 dist-test-slave-2sb1 krb5kdc[22644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880688, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:51:28.634014 22657 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 14:51:28.635238 22657 webserver.cc:460] Webserver started at http://127.0.98.62:39293/ using document root <none> and password file <none>
I20260619 14:51:28.635979 22657 fs_manager.cc:359] Metadata directory not provided
I20260619 14:51:28.636075 22657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:51:28.636335 22657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:51:28.638947 22657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880383132139-392-0/minicluster-data/master-0/data/instance:
uuid: "180c49198c88416a991a3eb11c449141"
format_stamp: "Formatted at 2026-06-19 14:51:28 on dist-test-slave-2sb1"
server_key: "8ceceef606a0e8ae194020ca0df75acf"
server_key_iv: "14a5c37053f21de5cbf2e5be9c1e832a"
server_key_version: "encryptionkey@0"
I20260619 14:51:28.639665 22657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880383132139-392-0/minicluster-data/master-0/wal/instance:
uuid: "180c49198c88416a991a3eb11c449141"
format_stamp: "Formatted at 2026-06-19 14:51:28 on dist-test-slave-2sb1"
server_key: "8ceceef606a0e8ae194020ca0df75acf"
server_key_iv: "14a5c37053f21de5cbf2e5be9c1e832a"
server_key_version: "encryptionkey@0"
I20260619 14:51:28.644435 22657 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260619 14:51:28.647733 22672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:28.649598 22657 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:51:28.649756 22657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880383132139-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880383132139-392-0/minicluster-data/master-0/wal
uuid: "180c49198c88416a991a3eb11c449141"
format_stamp: "Formatted at 2026-06-19 14:51:28 on dist-test-slave-2sb1"
server_key: "8ceceef606a0e8ae194020ca0df75acf"
server_key_iv: "14a5c37053f21de5cbf2e5be9c1e832a"
server_key_version: "encryptionkey@0"
I20260619 14:51:28.649941 22657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880383132139-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880383132139-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880383132139-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:51:28.671952 22657 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:51:28.676358 22657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:51:28.676580 22657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:51:28.684242 22657 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44253
I20260619 14:51:28.684595 22724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44253 every 8 connection(s)
I20260619 14:51:28.685595 22657 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:51:28.688827 22725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:51:28.693154 22725 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44253 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:51:28.695736 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 22657
I20260619 14:51:28.695868 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880383132139-392-0/minicluster-data/master-0/wal/instance
I20260619 14:51:28.696297 392 external_mini_cluster.cc:1428] Setting key a6c6c4dc2c8ac284336a0ae027dd70e5
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Jun 19 14:51:28 dist-test-slave-2sb1 krb5kdc[22644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880688, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:51:28.717516 22728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:28.696545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53254 (local address 127.0.98.62:44253)
0619 14:51:28.696858 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:28.696871 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:28.696918 (+ 47us) server_negotiation.cc:407] Connection header received
0619 14:51:28.698462 (+ 1544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:28.698471 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:28.698763 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:28.698989 (+ 226us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:28.699901 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:28.701453 (+ 1552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:28.702728 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:28.703416 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:28.703621 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:28.710548 (+ 6927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:28.710556 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:28.710569 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:28.710599 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:28.713165 (+ 2566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:28.713927 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:28.713942 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:28.713946 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:28.714138 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:28.714653 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:28.714659 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:28.714665 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:28.715573 (+ 908us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:28.715799 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:28.716299 (+ 500us) server_negotiation.cc:299] Negotiation successful
0619 14:51:28.716639 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":89,"server-negotiator.queue_time_us":207,"thread_start_us":157,"threads_started":1}
I20260619 14:51:28.717516 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:28.695571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44253 (local address 127.0.0.1:53254)
0619 14:51:28.696254 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:28.696320 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0619 14:51:28.697919 (+ 1599us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:51:28.699009 (+ 1090us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:51:28.699014 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:51:28.699376 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:51:28.699718 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:28.699733 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:28.701632 (+ 1899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:28.701635 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:28.702573 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:28.702584 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:28.703641 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:28.703645 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:28.703752 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:28.704610 (+ 858us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:51:28.704629 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:51:28.710228 (+ 5599us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:51:28.713337 (+ 3109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:28.713347 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:28.713357 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:28.713752 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:28.714265 (+ 513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:28.714268 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:28.714269 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:28.714417 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:28.715753 (+ 1336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:51:28.715759 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:51:28.715903 (+ 144us) client_negotiation.cc:769] Sending connection context
0619 14:51:28.716163 (+ 260us) client_negotiation.cc:240] Negotiation successful
0619 14:51:28.716487 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":217,"threads_started":1}
I20260619 14:51:28.720187 22725 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42607 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:51:28.721199 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:28.720855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42607 (local address 127.0.0.1:38914)
0619 14:51:28.721029 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:28.721121 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260619 14:51:28.721580 22674 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42607: connect: Connection refused (error 111)
I20260619 14:51:28.722699 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:28.722439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42607 (local address 127.0.0.1:38924)
0619 14:51:28.722578 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:28.722632 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260619 14:51:28.722922 22725 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:42607: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42607: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880383132139-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880383132139-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260619 14:51:28.731877 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880383132139-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880383132139-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880383132139-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880383132139-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:42607
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880383132139-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44253,127.0.98.61:42607,127.0.98.60:45231
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:51:28.772399 22725 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42607 } attempt: 1
I20260619 14:51:28.773491 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:28.773118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42607 (local address 127.0.0.1:38938)
0619 14:51:28.773307 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:28.773384 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260619 14:51:28.774333 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:28.774078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42607 (local address 127.0.0.1:38954)
0619 14:51:28.774204 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:28.774259 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260619 14:51:28.774509 22725 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:42607: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42607: connect: Connection refused (error 111)
I20260619 14:51:28.842733 22725 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42607 } attempt: 2
I20260619 14:51:28.843834 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:28.843387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42607 (local address 127.0.0.1:38968)
0619 14:51:28.843645 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:28.843724 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132}
I20260619 14:51:28.844732 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:28.844416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42607 (local address 127.0.0.1:38984)
0619 14:51:28.844557 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:28.844618 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260619 14:51:28.844964 22725 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:42607: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42607: connect: Connection refused (error 111)
W20260619 14:51:28.856273 22732 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 14:51:28.856753 22732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:51:28.856823 22732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:51:28.856928 22732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:51:28.861860 22732 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:51:28.861985 22732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:51:28.862030 22732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:51:28.862071 22732 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:51:28.862115 22732 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:51:28.868796 22732 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880383132139-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880383132139-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44253,127.0.98.61:42607,127.0.98.60:45231
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880383132139-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:42607
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880383132139-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22732
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880383132139-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:51:28.870131 22732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:51:28.871368 22732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:51:28.882808 22740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:51:28.882800 22738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:51:28.882800 22737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:51:28.883464 22732 server_base.cc:1034] running on GCE node
I20260619 14:51:28.884122 22732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:51:28.885859 22732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:51:28.887125 22732 hybrid_clock.cc:648] HybridClock initialized: now 1781880688887096 us; error 49 us; skew 500 ppm
Jun 19 14:51:28 dist-test-slave-2sb1 krb5kdc[22644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880688, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:51:28.891021 22732 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260619 14:51:28.892454 22732 webserver.cc:460] Webserver started at http://127.0.98.61:41185/ using document root <none> and password file <none>
I20260619 14:51:28.893214 22732 fs_manager.cc:359] Metadata directory not provided
I20260619 14:51:28.893335 22732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:51:28.893615 22732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:51:28.896112 22732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880383132139-392-0/minicluster-data/master-1/data/instance:
uuid: "fdf2df4996d64656b1c06a1152be08b1"
format_stamp: "Formatted at 2026-06-19 14:51:28 on dist-test-slave-2sb1"
server_key: "98bfca216b6971880e69a6662ef51f7d"
server_key_iv: "0a256efa9540fb382cebf868eeaa44a0"
server_key_version: "encryptionkey@0"
I20260619 14:51:28.896926 22732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880383132139-392-0/minicluster-data/master-1/wal/instance:
uuid: "fdf2df4996d64656b1c06a1152be08b1"
format_stamp: "Formatted at 2026-06-19 14:51:28 on dist-test-slave-2sb1"
server_key: "98bfca216b6971880e69a6662ef51f7d"
server_key_iv: "0a256efa9540fb382cebf868eeaa44a0"
server_key_version: "encryptionkey@0"
I20260619 14:51:28.901489 22732 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260619 14:51:28.904875 22747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:28.906490 22732 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260619 14:51:28.906674 22732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880383132139-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880383132139-392-0/minicluster-data/master-1/wal
uuid: "fdf2df4996d64656b1c06a1152be08b1"
format_stamp: "Formatted at 2026-06-19 14:51:28 on dist-test-slave-2sb1"
server_key: "98bfca216b6971880e69a6662ef51f7d"
server_key_iv: "0a256efa9540fb382cebf868eeaa44a0"
server_key_version: "encryptionkey@0"
I20260619 14:51:28.906819 22732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880383132139-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880383132139-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880383132139-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:51:28.921829 22732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:51:28.925920 22732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:51:28.926192 22732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:51:28.934720 22732 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:42607
I20260619 14:51:28.935146 22799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:42607 every 8 connection(s)
I20260619 14:51:28.935885 22732 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880383132139-392-0/minicluster-data/master-1/data/info.pb
I20260619 14:51:28.936168 22725 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42607 } attempt: 3
I20260619 14:51:28.939990 22800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:51:28.940184 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 22732
I20260619 14:51:28.940357 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880383132139-392-0/minicluster-data/master-1/wal/instance
I20260619 14:51:28.940706 392 external_mini_cluster.cc:1428] Setting key b295e00b41435ba224438c4c04df3557
I20260619 14:51:28.945257 22800 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44253 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 14:51:28 dist-test-slave-2sb1 krb5kdc[22644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880688, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260619 14:51:28.953325 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:28.936797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42607 (local address 127.0.0.1:60474)
0619 14:51:28.937086 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:28.937106 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 14:51:28.937229 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:51:28.940439 (+ 3210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:51:28.940443 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:51:28.940707 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:51:28.940891 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:28.940901 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:28.943476 (+ 2575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:28.943483 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:28.944472 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:28.944490 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:28.945702 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:28.945709 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:28.945859 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:28.946526 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:51:28.946549 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:51:28.948355 (+ 1806us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:51:28.950906 (+ 2551us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:28.950909 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:28.950912 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:28.951142 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:28.951534 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:28.951538 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:28.951540 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:28.951643 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:28.952515 (+ 872us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:51:28.952518 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:51:28.952637 (+ 119us) client_negotiation.cc:769] Sending connection context
0619 14:51:28.952798 (+ 161us) client_negotiation.cc:240] Negotiation successful
0619 14:51:28.953107 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Jun 19 14:51:28 dist-test-slave-2sb1 krb5kdc[22644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880688, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:51:28.954749 22802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:28.937845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60474 (local address 127.0.98.61:42607)
0619 14:51:28.938439 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:28.938459 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:28.938508 (+ 49us) server_negotiation.cc:407] Connection header received
0619 14:51:28.939466 (+ 958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:28.939492 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:28.940032 (+ 540us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:28.941148 (+ 1116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:28.941618 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:28.943305 (+ 1687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:28.944720 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:28.945516 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:28.945723 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:28.948616 (+ 2893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:28.948621 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:28.948635 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:28.948727 (+ 92us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:28.950763 (+ 2036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:28.951271 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:28.951281 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:28.951289 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:28.951427 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:28.951729 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:28.951750 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:28.951752 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:28.952381 (+ 629us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:28.952510 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:28.953000 (+ 490us) server_negotiation.cc:299] Negotiation successful
0619 14:51:28.953485 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":235,"threads_started":1}
I20260619 14:51:28.957396 22725 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45231 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:51:28.958259 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:28.957939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45231 (local address 127.0.0.1:56884)
0619 14:51:28.958085 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:28.958150 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260619 14:51:28.959563 22728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:28.947572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38594 (local address 127.0.98.62:44253)
0619 14:51:28.947724 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:28.947730 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:28.948206 (+ 476us) server_negotiation.cc:407] Connection header received
0619 14:51:28.948470 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:28.948474 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:28.948531 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:28.948590 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:28.949853 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:28.950483 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:28.951636 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:28.952036 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:28.952228 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:28.955368 (+ 3140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:28.955376 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:28.955383 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:28.955422 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:28.957379 (+ 1957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:28.957984 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:28.957991 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:28.957993 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:28.958094 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:28.958462 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:28.958465 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:28.958467 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:28.958690 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:28.958816 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:28.959235 (+ 419us) server_negotiation.cc:299] Negotiation successful
0619 14:51:28.959375 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 14:51:28.959621 22804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:28.947563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44253 (local address 127.0.0.1:38594)
0619 14:51:28.948064 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:28.948126 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0619 14:51:28.948327 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:51:28.948817 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:51:28.948824 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:51:28.949529 (+ 705us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:51:28.949690 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:28.949698 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:28.950622 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:28.950629 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:28.951467 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:28.951478 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:28.952194 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:28.952197 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:28.952331 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:28.953089 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:51:28.953104 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:51:28.955137 (+ 2033us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:51:28.957541 (+ 2404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:28.957552 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:28.957558 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:28.957856 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:28.958203 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:28.958207 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:28.958209 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:28.958309 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:28.958817 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:51:28.958825 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:51:28.958970 (+ 145us) client_negotiation.cc:769] Sending connection context
0619 14:51:28.959139 (+ 169us) client_negotiation.cc:240] Negotiation successful
0619 14:51:28.959384 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":166,"threads_started":1}
I20260619 14:51:28.959374 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:28.959004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45231 (local address 127.0.0.1:56892)
0619 14:51:28.959268 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:28.959313 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199}
W20260619 14:51:28.959841 22725 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45231: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45231: connect: Connection refused (error 111)
I20260619 14:51:28.961048 22800 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42607 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 14:51:28 dist-test-slave-2sb1 krb5kdc[22644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880688, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 14:51:28.972486 22802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:28.961683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60490 (local address 127.0.98.61:42607)
0619 14:51:28.961872 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:28.961879 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:28.961897 (+ 18us) server_negotiation.cc:407] Connection header received
0619 14:51:28.961938 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:28.961941 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:28.961992 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:28.962083 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:28.962520 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:28.963373 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:28.964280 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:28.964708 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:28.964816 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:28.968541 (+ 3725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:28.968546 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:28.968554 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:28.968579 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:28.970584 (+ 2005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:28.971030 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:28.971035 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:28.971037 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:28.971173 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:28.971455 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:28.971458 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:28.971460 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:28.971650 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:28.971777 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:28.972074 (+ 297us) server_negotiation.cc:299] Negotiation successful
0619 14:51:28.972238 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":52}
I20260619 14:51:28.972486 22804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:28.961575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42607 (local address 127.0.0.1:60490)
0619 14:51:28.961765 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:28.961779 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:51:28.961857 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:51:28.962107 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:51:28.962109 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:51:28.962264 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:51:28.962364 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:28.962403 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:28.963508 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:28.963511 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:28.964146 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:28.964156 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:28.964861 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:28.964865 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:28.964953 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:28.965500 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:51:28.965513 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:51:28.968356 (+ 2843us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:51:28.970707 (+ 2351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:28.970714 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:28.970716 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:28.970928 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:28.971273 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:28.971276 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:28.971278 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:28.971359 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:28.971763 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:51:28.971766 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:51:28.971895 (+ 129us) client_negotiation.cc:769] Sending connection context
0619 14:51:28.972028 (+ 133us) client_negotiation.cc:240] Negotiation successful
0619 14:51:28.972297 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":119}
I20260619 14:51:28.973671 22800 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45231 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:51:28.974748 22804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:28.974391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45231 (local address 127.0.0.1:56908)
0619 14:51:28.974545 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:28.974629 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260619 14:51:28.975382 22751 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45231: connect: Connection refused (error 111)
I20260619 14:51:28.976512 22804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:28.976235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45231 (local address 127.0.0.1:56918)
0619 14:51:28.976376 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:28.976449 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260619 14:51:28.976806 22800 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45231: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45231: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880383132139-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880383132139-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260619 14:51:28.977936 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880383132139-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880383132139-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880383132139-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880383132139-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:45231
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880383132139-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44253,127.0.98.61:42607,127.0.98.60:45231
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:51:28.991025 22725 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45231 } attempt: 1
I20260619 14:51:28.992142 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:28.991639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45231 (local address 127.0.0.1:56922)
0619 14:51:28.991992 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:28.992063 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260619 14:51:28.993134 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:28.992856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45231 (local address 127.0.0.1:56936)
0619 14:51:28.993005 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:28.993069 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260619 14:51:28.993369 22725 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45231: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45231: connect: Connection refused (error 111)
I20260619 14:51:29.026130 22800 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45231 } attempt: 1
I20260619 14:51:29.027136 22804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:29.026772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45231 (local address 127.0.0.1:56942)
0619 14:51:29.026963 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:29.027041 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260619 14:51:29.027951 22804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:29.027702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45231 (local address 127.0.0.1:56956)
0619 14:51:29.027825 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:29.027882 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260619 14:51:29.028441 22800 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45231: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45231: connect: Connection refused (error 111)
I20260619 14:51:29.068603 22725 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45231 } attempt: 2
I20260619 14:51:29.069612 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:29.069254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45231 (local address 127.0.0.1:56972)
0619 14:51:29.069405 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:29.069531 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260619 14:51:29.070549 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:29.070243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45231 (local address 127.0.0.1:56976)
0619 14:51:29.070428 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:29.070480 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
W20260619 14:51:29.070742 22725 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45231: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45231: connect: Connection refused (error 111)
I20260619 14:51:29.096669 22800 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45231 } attempt: 2
I20260619 14:51:29.097791 22804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:29.097363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45231 (local address 127.0.0.1:56980)
0619 14:51:29.097603 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:29.097681 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
I20260619 14:51:29.098803 22804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:29.098530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45231 (local address 127.0.0.1:56982)
0619 14:51:29.098662 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:29.098728 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260619 14:51:29.099028 22800 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45231: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45231: connect: Connection refused (error 111)
W20260619 14:51:29.106952 22807 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 14:51:29.107355 22807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:51:29.107414 22807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:51:29.107497 22807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:51:29.112279 22807 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:51:29.112424 22807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:51:29.112464 22807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:51:29.112528 22807 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:51:29.112588 22807 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:51:29.118610 22807 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880383132139-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880383132139-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44253,127.0.98.61:42607,127.0.98.60:45231
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880383132139-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:45231
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880383132139-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22807
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880383132139-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:51:29.119856 22807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:51:29.121047 22807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:51:29.131275 22812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:51:29.131348 22813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:51:29.131572 22807 server_base.cc:1034] running on GCE node
W20260619 14:51:29.131345 22815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:51:29.132323 22807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:51:29.133682 22807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:51:29.134928 22807 hybrid_clock.cc:648] HybridClock initialized: now 1781880689134914 us; error 33 us; skew 500 ppm
Jun 19 14:51:29 dist-test-slave-2sb1 krb5kdc[22644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880689, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:51:29.138228 22807 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260619 14:51:29.139649 22807 webserver.cc:460] Webserver started at http://127.0.98.60:41961/ using document root <none> and password file <none>
I20260619 14:51:29.140354 22807 fs_manager.cc:359] Metadata directory not provided
I20260619 14:51:29.140468 22807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:51:29.140794 22807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:51:29.143355 22807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880383132139-392-0/minicluster-data/master-2/data/instance:
uuid: "b9d4f045b09c4145950fc63c3f014149"
format_stamp: "Formatted at 2026-06-19 14:51:29 on dist-test-slave-2sb1"
server_key: "fb8e414cb9cdd4571e7f504387fc83eb"
server_key_iv: "334e306f4153fece1ea313ffd1d60ec2"
server_key_version: "encryptionkey@0"
I20260619 14:51:29.144081 22807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880383132139-392-0/minicluster-data/master-2/wal/instance:
uuid: "b9d4f045b09c4145950fc63c3f014149"
format_stamp: "Formatted at 2026-06-19 14:51:29 on dist-test-slave-2sb1"
server_key: "fb8e414cb9cdd4571e7f504387fc83eb"
server_key_iv: "334e306f4153fece1ea313ffd1d60ec2"
server_key_version: "encryptionkey@0"
I20260619 14:51:29.149191 22807 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260619 14:51:29.152522 22822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:29.154278 22807 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:51:29.154479 22807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880383132139-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880383132139-392-0/minicluster-data/master-2/wal
uuid: "b9d4f045b09c4145950fc63c3f014149"
format_stamp: "Formatted at 2026-06-19 14:51:29 on dist-test-slave-2sb1"
server_key: "fb8e414cb9cdd4571e7f504387fc83eb"
server_key_iv: "334e306f4153fece1ea313ffd1d60ec2"
server_key_version: "encryptionkey@0"
I20260619 14:51:29.154618 22807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880383132139-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880383132139-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880383132139-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:51:29.169986 22725 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45231 } attempt: 3
I20260619 14:51:29.171160 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:29.170649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45231 (local address 127.0.0.1:56998)
0619 14:51:29.170880 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:29.171072 (+ 192us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260619 14:51:29.171950 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:29.171715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45231 (local address 127.0.0.1:57004)
0619 14:51:29.171829 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:29.171882 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260619 14:51:29.172073 22725 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45231: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45231: connect: Connection refused (error 111)
I20260619 14:51:29.178082 22807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:51:29.182049 22807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:51:29.182297 22807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:51:29.190265 22800 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45231 } attempt: 3
I20260619 14:51:29.190665 22807 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:45231
I20260619 14:51:29.190985 22874 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:45231 every 8 connection(s)
I20260619 14:51:29.192001 22807 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880383132139-392-0/minicluster-data/master-2/data/info.pb
I20260619 14:51:29.196015 22876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:51:29.196190 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 22807
I20260619 14:51:29.196331 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781880383132139-392-0/minicluster-data/master-2/wal/instance
I20260619 14:51:29.196583 392 external_mini_cluster.cc:1428] Setting key d1a46b6693e7fe7d34557a69add6a9c1
Jun 19 14:51:29 dist-test-slave-2sb1 krb5kdc[22644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880688, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 14:51:29.201030 22876 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44253 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 14:51:29 dist-test-slave-2sb1 krb5kdc[22644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880688, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 14:51:29.207008 22804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:29.190916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45231 (local address 127.0.0.1:57010)
0619 14:51:29.191086 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:29.191106 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0619 14:51:29.191215 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:51:29.194775 (+ 3560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:51:29.194778 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:51:29.194982 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:51:29.195099 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:29.195106 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:29.197084 (+ 1978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:29.197088 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:29.198033 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:29.198048 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:29.199280 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:29.199284 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:29.199387 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:29.200022 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:51:29.200040 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:51:29.201819 (+ 1779us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:51:29.204920 (+ 3101us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:29.204924 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:29.204927 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:29.205198 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:29.205640 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:29.205644 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:29.205646 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:29.205753 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:29.206439 (+ 686us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:51:29.206443 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:51:29.206561 (+ 118us) client_negotiation.cc:769] Sending connection context
0619 14:51:29.206681 (+ 120us) client_negotiation.cc:240] Negotiation successful
0619 14:51:29.206837 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260619 14:51:29.208146 22875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:29.191983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57010 (local address 127.0.98.60:45231)
0619 14:51:29.192638 (+ 655us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:29.192686 (+ 48us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:29.193010 (+ 324us) server_negotiation.cc:407] Connection header received
0619 14:51:29.193917 (+ 907us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:29.193940 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:29.194391 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:29.194807 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:29.195230 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:29.196753 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:29.198178 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:29.199092 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:29.199307 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:29.202033 (+ 2726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:29.202040 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:29.202057 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:29.202090 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:29.204807 (+ 2717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:29.205323 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:29.205330 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:29.205336 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:29.205497 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:29.205865 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:29.205869 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:29.205870 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:29.206311 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:29.206459 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:29.206845 (+ 386us) server_negotiation.cc:299] Negotiation successful
0619 14:51:29.207181 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":284,"threads_started":1}
I20260619 14:51:29.210997 22879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:29.197678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57016 (local address 127.0.98.60:45231)
0619 14:51:29.198051 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:29.198060 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:29.198074 (+ 14us) server_negotiation.cc:407] Connection header received
0619 14:51:29.198113 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:29.198116 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:29.198168 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:29.198261 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:29.199334 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:29.200579 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:29.201825 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:29.202166 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:29.202291 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:29.206175 (+ 3884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:29.206179 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:29.206181 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:29.206206 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:29.208153 (+ 1947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:29.209281 (+ 1128us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:29.209286 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:29.209288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:29.209401 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:29.209740 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:29.209743 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:29.209745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:29.209992 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:29.210116 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:29.210604 (+ 488us) server_negotiation.cc:299] Negotiation successful
0619 14:51:29.210747 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"spinlock_wait_cycles":6912,"thread_start_us":172,"threads_started":1}
Jun 19 14:51:29 dist-test-slave-2sb1 krb5kdc[22644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880689, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:51:29.214322 22800 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdf2df4996d64656b1c06a1152be08b1: Bootstrap starting.
I20260619 14:51:29.216303 22728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:29.203413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38598 (local address 127.0.98.62:44253)
0619 14:51:29.203557 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:29.203564 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:29.203902 (+ 338us) server_negotiation.cc:407] Connection header received
0619 14:51:29.204117 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:29.204121 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:29.204170 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:29.204223 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:29.205169 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:29.205856 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:29.207853 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:29.208200 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:29.208338 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:29.212200 (+ 3862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:29.212204 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:29.212206 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:29.212234 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:29.214229 (+ 1995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:29.214845 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:29.214849 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:29.214850 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:29.214933 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:29.215238 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:29.215240 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:29.215241 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:29.215475 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:29.215555 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:29.216084 (+ 529us) server_negotiation.cc:299] Negotiation successful
0619 14:51:29.216188 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260619 14:51:29.216383 22880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:29.203326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44253 (local address 127.0.0.1:38598)
0619 14:51:29.203769 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:29.203821 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0619 14:51:29.203999 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:51:29.204364 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:51:29.204368 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:51:29.204852 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:51:29.205021 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:29.205029 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:29.205958 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:29.205961 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:29.207706 (+ 1745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:29.207719 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:29.208464 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:29.208468 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:29.208574 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:29.209357 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:51:29.209372 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:51:29.212011 (+ 2639us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:51:29.214370 (+ 2359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:29.214382 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:29.214389 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:29.214732 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:29.215035 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:29.215039 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:29.215041 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:29.215148 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:29.215593 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:51:29.215600 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:51:29.215755 (+ 155us) client_negotiation.cc:769] Sending connection context
0619 14:51:29.215938 (+ 183us) client_negotiation.cc:240] Negotiation successful
0619 14:51:29.216174 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"spinlock_wait_cycles":4480,"thread_start_us":162,"threads_started":1}
I20260619 14:51:29.217700 22800 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fdf2df4996d64656b1c06a1152be08b1: Neither blocks nor log segments found. Creating new log.
I20260619 14:51:29.217744 22876 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42607 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:51:29.218685 22800 log.cc:824] T 00000000000000000000000000000000 P fdf2df4996d64656b1c06a1152be08b1: Log is configured to *not* fsync() on all Append() calls
I20260619 14:51:29.222028 22800 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdf2df4996d64656b1c06a1152be08b1: No bootstrap required, opened a new log
Jun 19 14:51:29 dist-test-slave-2sb1 krb5kdc[22644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880689, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Jun 19 14:51:29 dist-test-slave-2sb1 krb5kdc[22644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880688, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:51:29.227016 22800 raft_consensus.cc:348] T 00000000000000000000000000000000 P fdf2df4996d64656b1c06a1152be08b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "180c49198c88416a991a3eb11c449141" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44253 } } peers { permanent_uuid: "fdf2df4996d64656b1c06a1152be08b1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42607 } } peers { permanent_uuid: "b9d4f045b09c4145950fc63c3f014149" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45231 } }
I20260619 14:51:29.227283 22800 raft_consensus.cc:729] T 00000000000000000000000000000000 P fdf2df4996d64656b1c06a1152be08b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdf2df4996d64656b1c06a1152be08b1, State: Initialized, Role: FOLLOWER
I20260619 14:51:29.227707 22800 consensus_queue.cc:260] T 00000000000000000000000000000000 P fdf2df4996d64656b1c06a1152be08b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "180c49198c88416a991a3eb11c449141" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44253 } } peers { permanent_uuid: "fdf2df4996d64656b1c06a1152be08b1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42607 } } peers { permanent_uuid: "b9d4f045b09c4145950fc63c3f014149" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45231 } }
I20260619 14:51:29.229835 22802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:29.218484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60506 (local address 127.0.98.61:42607)
0619 14:51:29.218607 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:29.218617 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:29.218694 (+ 77us) server_negotiation.cc:407] Connection header received
0619 14:51:29.218960 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:29.218963 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:29.219004 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:29.219072 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:29.219662 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:29.220211 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:29.221453 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:29.221788 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:29.221918 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:29.225190 (+ 3272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:29.225198 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:29.225204 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:29.225238 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:29.227517 (+ 2279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:29.228128 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:29.228133 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:29.228135 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:29.228235 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:29.228633 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:29.228637 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:29.228639 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:29.228959 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:29.229047 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:29.229493 (+ 446us) server_negotiation.cc:299] Negotiation successful
0619 14:51:29.229666 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 14:51:29.229924 22800 sys_catalog.cc:564] T 00000000000000000000000000000000 P fdf2df4996d64656b1c06a1152be08b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:51:29.230279 22728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:29.219322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38600 (local address 127.0.98.62:44253)
0619 14:51:29.219485 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:29.219492 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:29.219507 (+ 15us) server_negotiation.cc:407] Connection header received
0619 14:51:29.219548 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:29.219554 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:29.219605 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:29.219692 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:29.220356 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:29.220932 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:29.221931 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:29.222306 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:29.222400 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:29.225258 (+ 2858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:29.225264 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:29.225269 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:29.225294 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:29.227618 (+ 2324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:29.228115 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:29.228119 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:29.228121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:29.228222 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:29.228802 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:29.228806 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:29.228808 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:29.229241 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:29.229577 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:29.229810 (+ 233us) server_negotiation.cc:299] Negotiation successful
0619 14:51:29.230098 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260619 14:51:29.230695 22880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:29.218476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42607 (local address 127.0.0.1:60506)
0619 14:51:29.218626 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:29.218643 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 14:51:29.218748 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:51:29.219160 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:51:29.219166 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:51:29.219390 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:51:29.219523 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:29.219530 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:29.220358 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:29.220362 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:29.221281 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:29.221326 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:29.221927 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:29.221931 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:29.222036 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:29.222631 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:51:29.222647 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:51:29.225006 (+ 2359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:51:29.227668 (+ 2662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:29.227672 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:29.227675 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:29.227981 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:29.228390 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:29.228393 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:29.228396 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:29.228503 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:29.229132 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:51:29.229134 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:51:29.229245 (+ 111us) client_negotiation.cc:769] Sending connection context
0619 14:51:29.230340 (+ 1095us) client_negotiation.cc:240] Negotiation successful
0619 14:51:29.230516 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260619 14:51:29.231632 22876 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45231 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:51:29.232744 22882 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdf2df4996d64656b1c06a1152be08b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "180c49198c88416a991a3eb11c449141" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44253 } } peers { permanent_uuid: "fdf2df4996d64656b1c06a1152be08b1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42607 } } peers { permanent_uuid: "b9d4f045b09c4145950fc63c3f014149" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45231 } } }
I20260619 14:51:29.233560 22882 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdf2df4996d64656b1c06a1152be08b1 [sys.catalog]: This master's current role is: FOLLOWER
Jun 19 14:51:29 dist-test-slave-2sb1 krb5kdc[22644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880689, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 19 14:51:29 dist-test-slave-2sb1 krb5kdc[22644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880688, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 14:51:29.234933 22732 master_runner.cc:186] Error getting master registration for 127.0.98.62:44253: OK, instance_id { permanent_uuid: "180c49198c88416a991a3eb11c449141" instance_seqno: 1781880688681804 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 14:51:29.239651 22732 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 14:51:29.242297 22880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:29.232202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45231 (local address 127.0.0.1:57032)
0619 14:51:29.232458 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:29.232473 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:51:29.232561 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:51:29.232870 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:51:29.232874 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:51:29.233046 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:51:29.233128 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:29.233134 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:29.234156 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:29.234162 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:29.234780 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:29.234789 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:29.235412 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:29.235418 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:29.235534 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:29.236071 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:51:29.236087 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:51:29.237973 (+ 1886us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:51:29.240551 (+ 2578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:29.240554 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:29.240556 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:29.240859 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:29.241197 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:29.241199 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:29.241201 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:29.241280 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:29.241755 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:51:29.241757 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:51:29.241869 (+ 112us) client_negotiation.cc:769] Sending connection context
0619 14:51:29.241982 (+ 113us) client_negotiation.cc:240] Negotiation successful
0619 14:51:29.242133 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165}
I20260619 14:51:29.242969 22879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:29.232451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57032 (local address 127.0.98.60:45231)
0619 14:51:29.232586 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:29.232591 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:29.232606 (+ 15us) server_negotiation.cc:407] Connection header received
0619 14:51:29.232676 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:29.232681 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:29.232731 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:29.232815 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:29.233262 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:29.234033 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:29.234926 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:29.235269 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:29.235396 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:29.238139 (+ 2743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:29.238167 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:29.238171 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:29.238203 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:29.240385 (+ 2182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:29.240981 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:29.240985 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:29.240987 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:29.241088 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:29.241361 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:29.241363 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:29.241365 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:29.241634 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:29.241722 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:29.242702 (+ 980us) server_negotiation.cc:299] Negotiation successful
0619 14:51:29.242824 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260619 14:51:29.243114 22893 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fdf2df4996d64656b1c06a1152be08b1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:51:29.243189 22893 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:51:29.243741 22802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:29.231721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60520 (local address 127.0.98.61:42607)
0619 14:51:29.231831 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:29.231836 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:29.231848 (+ 12us) server_negotiation.cc:407] Connection header received
0619 14:51:29.231885 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:29.231890 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:29.231935 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:29.232011 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:29.232957 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:29.233663 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:29.234779 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:29.235097 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:29.235219 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:29.238798 (+ 3579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:29.238802 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:29.238805 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:29.238862 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:29.241101 (+ 2239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:29.241672 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:29.241677 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:29.241679 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:29.241769 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:29.242150 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:29.242154 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:29.242155 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:29.242375 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:29.242546 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:29.243444 (+ 898us) server_negotiation.cc:299] Negotiation successful
0619 14:51:29.243583 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260619 14:51:29.246750 22876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9d4f045b09c4145950fc63c3f014149: Bootstrap starting.
I20260619 14:51:29.248459 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:51:29.249922 22876 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b9d4f045b09c4145950fc63c3f014149: Neither blocks nor log segments found. Creating new log.
I20260619 14:51:29.249871 22699 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "180c49198c88416a991a3eb11c449141"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38600
I20260619 14:51:29.250938 22876 log.cc:824] T 00000000000000000000000000000000 P b9d4f045b09c4145950fc63c3f014149: Log is configured to *not* fsync() on all Append() calls
I20260619 14:51:29.253795 22876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9d4f045b09c4145950fc63c3f014149: No bootstrap required, opened a new log
I20260619 14:51:29.258296 22876 raft_consensus.cc:348] T 00000000000000000000000000000000 P b9d4f045b09c4145950fc63c3f014149 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "180c49198c88416a991a3eb11c449141" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44253 } } peers { permanent_uuid: "fdf2df4996d64656b1c06a1152be08b1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42607 } } peers { permanent_uuid: "b9d4f045b09c4145950fc63c3f014149" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45231 } }
I20260619 14:51:29.258617 22876 raft_consensus.cc:729] T 00000000000000000000000000000000 P b9d4f045b09c4145950fc63c3f014149 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9d4f045b09c4145950fc63c3f014149, State: Initialized, Role: FOLLOWER
I20260619 14:51:29.259011 22876 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9d4f045b09c4145950fc63c3f014149 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "180c49198c88416a991a3eb11c449141" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44253 } } peers { permanent_uuid: "fdf2df4996d64656b1c06a1152be08b1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42607 } } peers { permanent_uuid: "b9d4f045b09c4145950fc63c3f014149" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45231 } }
I20260619 14:51:29.259670 22896 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9d4f045b09c4145950fc63c3f014149 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "180c49198c88416a991a3eb11c449141" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44253 } } peers { permanent_uuid: "fdf2df4996d64656b1c06a1152be08b1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42607 } } peers { permanent_uuid: "b9d4f045b09c4145950fc63c3f014149" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45231 } } }
I20260619 14:51:29.259861 22896 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9d4f045b09c4145950fc63c3f014149 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:51:29.260497 22876 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9d4f045b09c4145950fc63c3f014149 [sys.catalog]: configured and running, proceeding with master startup.
W20260619 14:51:29.266974 22907 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b9d4f045b09c4145950fc63c3f014149: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:51:29.267084 22907 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:51:29.265246 22807 master_runner.cc:186] Error getting master registration for 127.0.98.62:44253: OK, instance_id { permanent_uuid: "180c49198c88416a991a3eb11c449141" instance_seqno: 1781880688681804 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 14:51:29.268638 22807 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 14:51:29.336216 22725 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45231 } attempt: 4
Jun 19 14:51:29 dist-test-slave-2sb1 krb5kdc[22644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880688, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 14:51:29.347822 22727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:29.336994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45231 (local address 127.0.0.1:57034)
0619 14:51:29.337194 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:29.337209 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0619 14:51:29.337316 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:51:29.337657 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:51:29.337660 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:51:29.337946 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:51:29.338066 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:29.338073 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:29.339178 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:29.339181 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:29.340238 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:29.340252 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:29.341071 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:29.341074 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:29.341166 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:29.341776 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:51:29.341792 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:51:29.343422 (+ 1630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:51:29.345953 (+ 2531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:29.345958 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:29.345960 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:29.346243 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:29.346596 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:29.346599 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:29.346601 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:29.346703 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:29.347245 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:51:29.347248 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:51:29.347353 (+ 105us) client_negotiation.cc:769] Sending connection context
0619 14:51:29.347455 (+ 102us) client_negotiation.cc:240] Negotiation successful
0619 14:51:29.347641 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123}
I20260619 14:51:29.348508 22879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:29.336988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57034 (local address 127.0.98.60:45231)
0619 14:51:29.337145 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:29.337149 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:29.337276 (+ 127us) server_negotiation.cc:407] Connection header received
0619 14:51:29.337444 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:29.337448 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:29.337503 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:29.337609 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:29.338207 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:29.339027 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:29.340407 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:29.340807 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:29.340908 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:29.343637 (+ 2729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:29.343641 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:29.343644 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:29.343676 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:29.345720 (+ 2044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:29.346368 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:29.346372 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:29.346374 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:29.346460 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:29.346819 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:29.346824 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:29.346826 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:29.347099 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:29.347196 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:29.348148 (+ 952us) server_negotiation.cc:299] Negotiation successful
0619 14:51:29.348299 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260619 14:51:29.352411 22725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 180c49198c88416a991a3eb11c449141: Bootstrap starting.
I20260619 14:51:29.355908 22725 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 180c49198c88416a991a3eb11c449141: Neither blocks nor log segments found. Creating new log.
I20260619 14:51:29.356947 22725 log.cc:824] T 00000000000000000000000000000000 P 180c49198c88416a991a3eb11c449141: Log is configured to *not* fsync() on all Append() calls
I20260619 14:51:29.359478 22725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 180c49198c88416a991a3eb11c449141: No bootstrap required, opened a new log
I20260619 14:51:29.362751 22725 raft_consensus.cc:348] T 00000000000000000000000000000000 P 180c49198c88416a991a3eb11c449141 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "180c49198c88416a991a3eb11c449141" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44253 } } peers { permanent_uuid: "fdf2df4996d64656b1c06a1152be08b1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42607 } } peers { permanent_uuid: "b9d4f045b09c4145950fc63c3f014149" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45231 } }
I20260619 14:51:29.363085 22725 raft_consensus.cc:729] T 00000000000000000000000000000000 P 180c49198c88416a991a3eb11c449141 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 180c49198c88416a991a3eb11c449141, State: Initialized, Role: FOLLOWER
I20260619 14:51:29.363444 22725 consensus_queue.cc:260] T 00000000000000000000000000000000 P 180c49198c88416a991a3eb11c449141 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "180c49198c88416a991a3eb11c449141" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44253 } } peers { permanent_uuid: "fdf2df4996d64656b1c06a1152be08b1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42607 } } peers { permanent_uuid: "b9d4f045b09c4145950fc63c3f014149" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45231 } }
I20260619 14:51:29.364184 22910 sys_catalog.cc:455] T 00000000000000000000000000000000 P 180c49198c88416a991a3eb11c449141 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "180c49198c88416a991a3eb11c449141" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44253 } } peers { permanent_uuid: "fdf2df4996d64656b1c06a1152be08b1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42607 } } peers { permanent_uuid: "b9d4f045b09c4145950fc63c3f014149" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45231 } } }
I20260619 14:51:29.364494 22910 sys_catalog.cc:458] T 00000000000000000000000000000000 P 180c49198c88416a991a3eb11c449141 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:51:29.365252 22725 sys_catalog.cc:564] T 00000000000000000000000000000000 P 180c49198c88416a991a3eb11c449141 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:51:29.365320 22699 raft_consensus.cc:482] T 00000000000000000000000000000000 P 180c49198c88416a991a3eb11c449141 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 14:51:29.365444 22699 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 180c49198c88416a991a3eb11c449141 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:51:29.366796 22699 raft_consensus.cc:504] T 00000000000000000000000000000000 P 180c49198c88416a991a3eb11c449141 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "180c49198c88416a991a3eb11c449141" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44253 } } peers { permanent_uuid: "fdf2df4996d64656b1c06a1152be08b1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42607 } } peers { permanent_uuid: "b9d4f045b09c4145950fc63c3f014149" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45231 } }
I20260619 14:51:29.367837 22699 leader_election.cc:288] T 00000000000000000000000000000000 P 180c49198c88416a991a3eb11c449141 [CANDIDATE]: Term 1 election: Requested vote from peers fdf2df4996d64656b1c06a1152be08b1 (127.0.98.61:42607), b9d4f045b09c4145950fc63c3f014149 (127.0.98.60:45231)
I20260619 14:51:29.369041 22774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "180c49198c88416a991a3eb11c449141" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "fdf2df4996d64656b1c06a1152be08b1"
I20260619 14:51:29.369328 22774 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fdf2df4996d64656b1c06a1152be08b1 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:51:29.369975 22849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "180c49198c88416a991a3eb11c449141" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "b9d4f045b09c4145950fc63c3f014149"
I20260619 14:51:29.370209 22849 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b9d4f045b09c4145950fc63c3f014149 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:51:29.370739 22774 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fdf2df4996d64656b1c06a1152be08b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 180c49198c88416a991a3eb11c449141 in term 1.
I20260619 14:51:29.371270 22674 leader_election.cc:302] T 00000000000000000000000000000000 P 180c49198c88416a991a3eb11c449141 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 180c49198c88416a991a3eb11c449141, fdf2df4996d64656b1c06a1152be08b1; no voters:
I20260619 14:51:29.371541 22849 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b9d4f045b09c4145950fc63c3f014149 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 180c49198c88416a991a3eb11c449141 in term 1.
I20260619 14:51:29.374173 22910 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 180c49198c88416a991a3eb11c449141 [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:51:29.374495 22910 raft_consensus.cc:686] T 00000000000000000000000000000000 P 180c49198c88416a991a3eb11c449141 [term 1 LEADER]: Becoming Leader. State: Replica: 180c49198c88416a991a3eb11c449141, State: Running, Role: LEADER
I20260619 14:51:29.374905 22910 consensus_queue.cc:237] T 00000000000000000000000000000000 P 180c49198c88416a991a3eb11c449141 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "180c49198c88416a991a3eb11c449141" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44253 } } peers { permanent_uuid: "fdf2df4996d64656b1c06a1152be08b1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42607 } } peers { permanent_uuid: "b9d4f045b09c4145950fc63c3f014149" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45231 } }
I20260619 14:51:29.377022 22910 sys_catalog.cc:455] T 00000000000000000000000000000000 P 180c49198c88416a991a3eb11c449141 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 180c49198c88416a991a3eb11c449141. Latest consensus state: current_term: 1 leader_uuid: "180c49198c88416a991a3eb11c449141" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "180c49198c88416a991a3eb11c449141" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44253 } } peers { permanent_uuid: "fdf2df4996d64656b1c06a1152be08b1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42607 } } peers { permanent_uuid: "b9d4f045b09c4145950fc63c3f014149" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45231 } } }
I20260619 14:51:29.377213 22910 sys_catalog.cc:458] T 00000000000000000000000000000000 P 180c49198c88416a991a3eb11c449141 [sys.catalog]: This master's current role is: LEADER
I20260619 14:51:29.380008 22657 master_runner.cc:263] No leader master found from master 180c49198c88416a991a3eb11c449141
I20260619 14:51:29.380095 22657 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 14:51:29.391187 22728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:29.373794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38606 (local address 127.0.98.62:44253)
0619 14:51:29.373924 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:29.373930 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:29.380568 (+ 6638us) server_negotiation.cc:407] Connection header received
0619 14:51:29.380869 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:29.380872 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:29.381020 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:29.381082 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:29.382183 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:29.383046 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:29.384322 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:29.384761 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:29.384848 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:29.386991 (+ 2143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:29.386997 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:29.387003 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:29.387040 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:29.388935 (+ 1895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:29.389606 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:29.389610 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:29.389612 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:29.389733 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:29.390098 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:29.390101 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:29.390103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:29.390350 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:29.390478 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:29.390877 (+ 399us) server_negotiation.cc:299] Negotiation successful
0619 14:51:29.391010 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260619 14:51:29.380079 22921 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 180c49198c88416a991a3eb11c449141: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:51:29.392764 22921 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:51:29.392874 22928 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:51:29.393869 22928 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:51:29.403574 22879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:29.380608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57048 (local address 127.0.98.60:45231)
0619 14:51:29.380767 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:29.380772 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:29.393046 (+ 12274us) server_negotiation.cc:407] Connection header received
0619 14:51:29.393239 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:29.393242 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:29.393301 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:29.393358 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:29.394320 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:29.395035 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:29.396366 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:29.396753 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:29.396849 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:29.399206 (+ 2357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:29.399211 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:29.399214 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:29.399253 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:29.401287 (+ 2034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:29.401938 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:29.401943 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:29.401946 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:29.402054 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:29.402442 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:29.402447 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:29.402449 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:29.402698 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:29.402812 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:29.403235 (+ 423us) server_negotiation.cc:299] Negotiation successful
0619 14:51:29.403395 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260619 14:51:29.407958 22802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:29.377825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60534 (local address 127.0.98.61:42607)
0619 14:51:29.377942 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:29.377947 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:29.389012 (+ 11065us) server_negotiation.cc:407] Connection header received
0619 14:51:29.389230 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:29.389234 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:29.389288 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:29.389342 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:29.390520 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:29.391260 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:29.400991 (+ 9731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:29.401343 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:29.401438 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:29.403395 (+ 1957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:29.403400 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:29.403403 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:29.403439 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:29.405778 (+ 2339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:29.406358 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:29.406365 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:29.406370 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:29.406465 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:29.406781 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:29.406785 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:29.406787 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:29.407046 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:29.407202 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:29.407603 (+ 401us) server_negotiation.cc:299] Negotiation successful
0619 14:51:29.407776 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260619 14:51:29.411761 22849 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b9d4f045b09c4145950fc63c3f014149 [term 1 FOLLOWER]: Refusing update from remote peer 180c49198c88416a991a3eb11c449141: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:51:29.413311 22926 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 180c49198c88416a991a3eb11c449141 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d4f045b09c4145950fc63c3f014149" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260619 14:51:29.413813 22774 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fdf2df4996d64656b1c06a1152be08b1 [term 1 FOLLOWER]: Refusing update from remote peer 180c49198c88416a991a3eb11c449141: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:51:29.419395 22926 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 180c49198c88416a991a3eb11c449141 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdf2df4996d64656b1c06a1152be08b1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:51:29.419636 22896 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9d4f045b09c4145950fc63c3f014149 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 180c49198c88416a991a3eb11c449141. Latest consensus state: current_term: 1 leader_uuid: "180c49198c88416a991a3eb11c449141" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "180c49198c88416a991a3eb11c449141" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44253 } } peers { permanent_uuid: "fdf2df4996d64656b1c06a1152be08b1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42607 } } peers { permanent_uuid: "b9d4f045b09c4145950fc63c3f014149" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45231 } } }
I20260619 14:51:29.419761 22896 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9d4f045b09c4145950fc63c3f014149 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:51:29.424738 22882 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdf2df4996d64656b1c06a1152be08b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 180c49198c88416a991a3eb11c449141. Latest consensus state: current_term: 1 leader_uuid: "180c49198c88416a991a3eb11c449141" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "180c49198c88416a991a3eb11c449141" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44253 } } peers { permanent_uuid: "fdf2df4996d64656b1c06a1152be08b1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42607 } } peers { permanent_uuid: "b9d4f045b09c4145950fc63c3f014149" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45231 } } }
I20260619 14:51:29.424925 22882 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdf2df4996d64656b1c06a1152be08b1 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:51:29.427896 22926 sys_catalog.cc:455] T 00000000000000000000000000000000 P 180c49198c88416a991a3eb11c449141 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "180c49198c88416a991a3eb11c449141" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "180c49198c88416a991a3eb11c449141" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44253 } } peers { permanent_uuid: "fdf2df4996d64656b1c06a1152be08b1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42607 } } peers { permanent_uuid: "b9d4f045b09c4145950fc63c3f014149" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45231 } } }
I20260619 14:51:29.428056 22926 sys_catalog.cc:458] T 00000000000000000000000000000000 P 180c49198c88416a991a3eb11c449141 [sys.catalog]: This master's current role is: LEADER
W20260619 14:51:29.428438 22753 tablet.cc:2369] T 00000000000000000000000000000000 P fdf2df4996d64656b1c06a1152be08b1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260619 14:51:29.430397 22896 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9d4f045b09c4145950fc63c3f014149 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "180c49198c88416a991a3eb11c449141" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "180c49198c88416a991a3eb11c449141" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44253 } } peers { permanent_uuid: "fdf2df4996d64656b1c06a1152be08b1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42607 } } peers { permanent_uuid: "b9d4f045b09c4145950fc63c3f014149" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45231 } } }
I20260619 14:51:29.430567 22896 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9d4f045b09c4145950fc63c3f014149 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:51:29.434118 22934 mvcc.cc:205] Tried to move back new op lower bound from 7298583303821537280 to 7298583303683903488. Current Snapshot: MvccSnapshot[applied={T|T < 7298583303821537280}]
I20260619 14:51:29.435438 22910 sys_catalog.cc:455] T 00000000000000000000000000000000 P 180c49198c88416a991a3eb11c449141 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "180c49198c88416a991a3eb11c449141" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "180c49198c88416a991a3eb11c449141" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44253 } } peers { permanent_uuid: "fdf2df4996d64656b1c06a1152be08b1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42607 } } peers { permanent_uuid: "b9d4f045b09c4145950fc63c3f014149" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45231 } } }
I20260619 14:51:29.435577 22910 sys_catalog.cc:458] T 00000000000000000000000000000000 P 180c49198c88416a991a3eb11c449141 [sys.catalog]: This master's current role is: LEADER
I20260619 14:51:29.436808 22882 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdf2df4996d64656b1c06a1152be08b1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "180c49198c88416a991a3eb11c449141" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "180c49198c88416a991a3eb11c449141" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44253 } } peers { permanent_uuid: "fdf2df4996d64656b1c06a1152be08b1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42607 } } peers { permanent_uuid: "b9d4f045b09c4145950fc63c3f014149" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45231 } } }
I20260619 14:51:29.436980 22882 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdf2df4996d64656b1c06a1152be08b1 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:51:29.438982 22928 catalog_manager.cc:1347] Generated new cluster ID: 7842cdc80b6f464bb878c7aafd909534
I20260619 14:51:29.439064 22928 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:51:29.467585 22928 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:51:29.469300 22928 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:51:29.479776 22928 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 180c49198c88416a991a3eb11c449141: Generated new TSK 0
I20260619 14:51:29.480774 22928 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 14:51:29.511508 22879 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:51:29.509870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57064 (local address 127.0.98.60:45231)
0619 14:51:29.510002 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:29.510006 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:29.510263 (+ 257us) server_negotiation.cc:407] Connection header received
0619 14:51:29.510501 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:29.510506 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:29.510571 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:29.510630 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:29.511358 (+ 728us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57064: BlockingRecv error: recv got EOF from 127.0.0.1:57064 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260619 14:51:29.511492 22802 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:51:29.509628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60536 (local address 127.0.98.61:42607)
0619 14:51:29.509977 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:29.509983 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:29.510252 (+ 269us) server_negotiation.cc:407] Connection header received
0619 14:51:29.510515 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:29.510519 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:29.510572 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:29.510629 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:29.511342 (+ 713us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60536: BlockingRecv error: recv got EOF from 127.0.0.1:60536 (error 108)
Metrics: {"server-negotiator.queue_time_us":249}
W20260619 14:51:29.511690 22728 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:51:29.509764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38612 (local address 127.0.98.62:44253)
0619 14:51:29.510036 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:29.510040 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:29.510567 (+ 527us) server_negotiation.cc:407] Connection header received
0619 14:51:29.510761 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:29.510765 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:29.510854 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:29.510937 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:29.511568 (+ 631us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38612: BlockingRecv error: recv got EOF from 127.0.0.1:38612 (error 108)
Metrics: {"server-negotiator.queue_time_us":196}
W20260619 14:51:29.515558 22802 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:51:29.514246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60540 (local address 127.0.98.61:42607)
0619 14:51:29.514407 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:29.514420 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:29.514659 (+ 239us) server_negotiation.cc:407] Connection header received
0619 14:51:29.514757 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:29.514760 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:29.514830 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:29.514914 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:29.515438 (+ 524us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60540: BlockingRecv error: recv got EOF from 127.0.0.1:60540 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260619 14:51:29.520363 22802 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:51:29.519233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60550 (local address 127.0.98.61:42607)
0619 14:51:29.519378 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:29.519383 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:29.519399 (+ 16us) server_negotiation.cc:407] Connection header received
0619 14:51:29.519454 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:29.519457 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:29.519517 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:29.519609 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:29.520234 (+ 625us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60550: BlockingRecv error: recv got EOF from 127.0.0.1:60550 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260619 14:51:29.526183 22802 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:51:29.525071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60558 (local address 127.0.98.61:42607)
0619 14:51:29.525226 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:29.525232 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:29.525248 (+ 16us) server_negotiation.cc:407] Connection header received
0619 14:51:29.525305 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:29.525308 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:29.525359 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:29.525449 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:29.526067 (+ 618us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60558: BlockingRecv error: recv got EOF from 127.0.0.1:60558 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260619 14:51:29.534067 22802 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:51:29.532710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60570 (local address 127.0.98.61:42607)
0619 14:51:29.532924 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:29.532929 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:29.533036 (+ 107us) server_negotiation.cc:407] Connection header received
0619 14:51:29.533161 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:29.533164 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:29.533231 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:29.533364 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:29.533933 (+ 569us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60570: BlockingRecv error: recv got EOF from 127.0.0.1:60570 (error 108)
Metrics: {"server-negotiator.queue_time_us":108}
W20260619 14:51:29.546026 22802 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:51:29.544566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60582 (local address 127.0.98.61:42607)
0619 14:51:29.544776 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:29.544782 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:29.545014 (+ 232us) server_negotiation.cc:407] Connection header received
0619 14:51:29.545221 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:29.545228 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:29.545296 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:29.545413 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:29.545887 (+ 474us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60582: BlockingRecv error: recv got EOF from 127.0.0.1:60582 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260619 14:51:29.565994 22802 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:51:29.564529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60584 (local address 127.0.98.61:42607)
0619 14:51:29.564730 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:29.564736 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:29.564918 (+ 182us) server_negotiation.cc:407] Connection header received
0619 14:51:29.565062 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:29.565066 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:29.565136 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:29.565250 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:29.565856 (+ 606us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60584: BlockingRecv error: recv got EOF from 127.0.0.1:60584 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260619 14:51:29.602057 22802 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:51:29.600688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60596 (local address 127.0.98.61:42607)
0619 14:51:29.600838 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:29.600844 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:29.601025 (+ 181us) server_negotiation.cc:407] Connection header received
0619 14:51:29.601221 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:29.601224 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:29.601290 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:29.601458 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:29.601938 (+ 480us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60596: BlockingRecv error: recv got EOF from 127.0.0.1:60596 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260619 14:51:29.670323 22802 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:51:29.669016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60608 (local address 127.0.98.61:42607)
0619 14:51:29.669167 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:29.669171 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:29.669187 (+ 16us) server_negotiation.cc:407] Connection header received
0619 14:51:29.669407 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:29.669411 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:29.669498 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:29.669586 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:29.670214 (+ 628us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60608: BlockingRecv error: recv got EOF from 127.0.0.1:60608 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260619 14:51:29.802131 22802 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:51:29.801108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60618 (local address 127.0.98.61:42607)
0619 14:51:29.801262 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:29.801265 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:29.801280 (+ 15us) server_negotiation.cc:407] Connection header received
0619 14:51:29.801336 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:29.801339 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:29.801394 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:29.801482 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:29.802031 (+ 549us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60618: BlockingRecv error: recv got EOF from 127.0.0.1:60618 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260619 14:51:30.062137 22802 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:51:30.060790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60622 (local address 127.0.98.61:42607)
0619 14:51:30.061070 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:30.061076 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:30.061093 (+ 17us) server_negotiation.cc:407] Connection header received
0619 14:51:30.061266 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:30.061271 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:30.061327 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:30.061424 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:30.061990 (+ 566us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60622: BlockingRecv error: recv got EOF from 127.0.0.1:60622 (error 108)
Metrics: {"server-negotiator.queue_time_us":178}
I20260619 14:51:30.244846 22893 catalog_manager.cc:1259] Loaded cluster ID: 7842cdc80b6f464bb878c7aafd909534
I20260619 14:51:30.244939 22893 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fdf2df4996d64656b1c06a1152be08b1: loading cluster ID for follower catalog manager: success
I20260619 14:51:30.246711 22893 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fdf2df4996d64656b1c06a1152be08b1: acquiring CA information for follower catalog manager: success
I20260619 14:51:30.247951 22893 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fdf2df4996d64656b1c06a1152be08b1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 14:51:30.268580 22907 catalog_manager.cc:1259] Loaded cluster ID: 7842cdc80b6f464bb878c7aafd909534
I20260619 14:51:30.268694 22907 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b9d4f045b09c4145950fc63c3f014149: loading cluster ID for follower catalog manager: success
I20260619 14:51:30.270359 22907 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b9d4f045b09c4145950fc63c3f014149: acquiring CA information for follower catalog manager: success
I20260619 14:51:30.271559 22907 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b9d4f045b09c4145950fc63c3f014149: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 14:51:30.580950 23006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:30.576755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60638 (local address 127.0.98.61:42607)
0619 14:51:30.577150 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:30.577153 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:30.577166 (+ 13us) server_negotiation.cc:407] Connection header received
0619 14:51:30.577218 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:30.577220 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:30.577361 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:30.577495 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:51:30.577778 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:30.578327 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:30.579256 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:30.579493 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:30.579613 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:30.579970 (+ 357us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:51:30.580446 (+ 476us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:51:30.580602 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:30.580703 (+ 101us) server_negotiation.cc:299] Negotiation successful
0619 14:51:30.580779 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":79,"threads_started":1}
I20260619 14:51:30.589955 23012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:30.585076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57076 (local address 127.0.98.60:45231)
0619 14:51:30.585363 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:30.585367 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:30.585383 (+ 16us) server_negotiation.cc:407] Connection header received
0619 14:51:30.585470 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:30.585473 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:30.585595 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:30.585709 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:51:30.586045 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:30.586736 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:30.587831 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:30.588107 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:30.588209 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:30.588738 (+ 529us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:51:30.589333 (+ 595us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:51:30.589445 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:30.589660 (+ 215us) server_negotiation.cc:299] Negotiation successful
0619 14:51:30.589725 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":89,"threads_started":1}
I20260619 14:51:30.599085 23012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:30.594137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57086 (local address 127.0.98.60:45231)
0619 14:51:30.594274 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:30.594278 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:30.594561 (+ 283us) server_negotiation.cc:407] Connection header received
0619 14:51:30.594749 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:30.594751 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:30.594797 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:30.594888 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:51:30.595174 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:30.595942 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:30.597599 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:30.597976 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:30.598084 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:30.598564 (+ 480us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:51:30.598648 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:51:30.598694 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:30.598864 (+ 170us) server_negotiation.cc:299] Negotiation successful
0619 14:51:30.598933 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 14:51:30.599097 23006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:30.594149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60640 (local address 127.0.98.61:42607)
0619 14:51:30.594304 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:30.594309 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:30.594326 (+ 17us) server_negotiation.cc:407] Connection header received
0619 14:51:30.594487 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:30.594491 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:30.594566 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:30.594656 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:51:30.595070 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:30.595941 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:30.597606 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:30.597981 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:30.598127 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:30.598510 (+ 383us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:51:30.598598 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:51:30.598692 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:30.598851 (+ 159us) server_negotiation.cc:299] Negotiation successful
0619 14:51:30.598920 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260619 14:51:30.599771 23019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:30.593865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38614 (local address 127.0.98.62:44253)
0619 14:51:30.594493 (+ 628us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:30.594498 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:30.594513 (+ 15us) server_negotiation.cc:407] Connection header received
0619 14:51:30.594720 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:30.594724 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:30.594931 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:30.595071 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:51:30.595337 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:30.595925 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:30.597713 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:30.598074 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:30.598167 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:30.598610 (+ 443us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:51:30.599034 (+ 424us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:51:30.599122 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:30.599543 (+ 421us) server_negotiation.cc:299] Negotiation successful
0619 14:51:30.599618 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":73,"threads_started":1}
I20260619 14:51:30.604290 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 22657
I20260619 14:51:30.611464 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 22732
I20260619 14:51:30.618708 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 22807
2026-06-19T14:51:30Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3404 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880383132139-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 19 14:51:30 dist-test-slave-2sb1 krb5kdc[23023](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 19 14:51:30 dist-test-slave-2sb1 krb5kdc[23023](info): set up 2 sockets
Jun 19 14:51:30 dist-test-slave-2sb1 krb5kdc[23023](info): commencing operation
krb5kdc: starting...
W20260619 14:51:31.737573 392 mini_kdc.cc:121] Time spent starting KDC: real 1.095s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880383132139-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880383132139-392-0/krb5kdc/test-user.keytab.
Jun 19 14:51:31 dist-test-slave-2sb1 krb5kdc[23023](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880691, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-19T14:51:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-19T14:51:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260619 14:51:31.880815 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880383132139-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880383132139-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880383132139-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36421
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36421,127.0.98.61:45193,127.0.98.60:33439
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260619 14:51:32.002036 23036 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 14:51:32.002363 23036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:51:32.002389 23036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:51:32.002434 23036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:51:32.006707 23036 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:51:32.006798 23036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:51:32.006819 23036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:51:32.006837 23036 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:51:32.006853 23036 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:51:32.012791 23036 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880383132139-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880383132139-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36421,127.0.98.61:45193,127.0.98.60:33439
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880383132139-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23036
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880383132139-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:51:32.013952 23036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:51:32.014956 23036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:51:32.025035 23042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:51:32.025050 23041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:51:32.025111 23036 server_base.cc:1034] running on GCE node
W20260619 14:51:32.025064 23044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:51:32.026115 23036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:51:32.027386 23036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:51:32.028633 23036 hybrid_clock.cc:648] HybridClock initialized: now 1781880692028615 us; error 36 us; skew 500 ppm
Jun 19 14:51:32 dist-test-slave-2sb1 krb5kdc[23023](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880692, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:51:32.031905 23036 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260619 14:51:32.032970 23036 webserver.cc:460] Webserver started at http://127.0.98.62:35223/ using document root <none> and password file <none>
I20260619 14:51:32.033605 23036 fs_manager.cc:359] Metadata directory not provided
I20260619 14:51:32.033670 23036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:51:32.033852 23036 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:51:32.036006 23036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880383132139-392-0/minicluster-data/master-0/data/instance:
uuid: "a7e69f99b4a94fcb8376d048395019ba"
format_stamp: "Formatted at 2026-06-19 14:51:32 on dist-test-slave-2sb1"
server_key: "6f9b1cc007accb79622dda54fd46d9ce"
server_key_iv: "551612838515e289418cafcc8ebcd742"
server_key_version: "encryptionkey@0"
I20260619 14:51:32.036569 23036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880383132139-392-0/minicluster-data/master-0/wal/instance:
uuid: "a7e69f99b4a94fcb8376d048395019ba"
format_stamp: "Formatted at 2026-06-19 14:51:32 on dist-test-slave-2sb1"
server_key: "6f9b1cc007accb79622dda54fd46d9ce"
server_key_iv: "551612838515e289418cafcc8ebcd742"
server_key_version: "encryptionkey@0"
I20260619 14:51:32.041517 23036 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260619 14:51:32.044596 23051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:32.046470 23036 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260619 14:51:32.046648 23036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880383132139-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880383132139-392-0/minicluster-data/master-0/wal
uuid: "a7e69f99b4a94fcb8376d048395019ba"
format_stamp: "Formatted at 2026-06-19 14:51:32 on dist-test-slave-2sb1"
server_key: "6f9b1cc007accb79622dda54fd46d9ce"
server_key_iv: "551612838515e289418cafcc8ebcd742"
server_key_version: "encryptionkey@0"
I20260619 14:51:32.046803 23036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880383132139-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880383132139-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880383132139-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:51:32.067800 23036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:51:32.072124 23036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:51:32.072363 23036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:51:32.080753 23036 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36421
I20260619 14:51:32.081051 23103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36421 every 8 connection(s)
I20260619 14:51:32.081946 23036 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880383132139-392-0/minicluster-data/master-0/data/info.pb
I20260619 14:51:32.085022 23104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:51:32.088966 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 23036
I20260619 14:51:32.089110 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880383132139-392-0/minicluster-data/master-0/wal/instance
I20260619 14:51:32.089534 392 external_mini_cluster.cc:1428] Setting key 45b136ea2d86e1534807f07ed76cf3e4
I20260619 14:51:32.089314 23104 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36421 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Jun 19 14:51:32 dist-test-slave-2sb1 krb5kdc[23023](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880692, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:51:32.110477 23108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:32.092474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36421 (local address 127.0.0.1:47528)
0619 14:51:32.093290 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:32.093330 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0619 14:51:32.095061 (+ 1731us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:51:32.096056 (+ 995us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:51:32.096061 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:51:32.096553 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:51:32.096905 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:32.096924 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:32.098176 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:32.098180 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:32.098955 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:32.098964 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:32.099651 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:32.099655 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:32.099762 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:32.100598 (+ 836us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:51:32.100615 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:51:32.103638 (+ 3023us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:51:32.106584 (+ 2946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:32.106599 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:32.106615 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:32.107055 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:32.107531 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:32.107537 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:32.107541 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:32.107722 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:32.108544 (+ 822us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:51:32.108554 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:51:32.108812 (+ 258us) client_negotiation.cc:769] Sending connection context
0619 14:51:32.109041 (+ 229us) client_negotiation.cc:240] Negotiation successful
0619 14:51:32.109570 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"mutex_wait_us":142,"spinlock_wait_cycles":519552,"thread_start_us":374,"threads_started":1}
I20260619 14:51:32.110483 23107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:32.092478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47528 (local address 127.0.98.62:36421)
0619 14:51:32.093134 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:32.093154 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:32.093455 (+ 301us) server_negotiation.cc:407] Connection header received
0619 14:51:32.095585 (+ 2130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:32.095592 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:32.095844 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:32.096061 (+ 217us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:32.097115 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:32.098032 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:32.099086 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:32.099477 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:32.099666 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:32.103866 (+ 4200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:32.103882 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:32.103903 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:32.103947 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:32.106414 (+ 2467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:32.107257 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:32.107263 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:32.107264 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:32.107393 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:32.107958 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:32.107961 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:32.107962 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:32.108397 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:32.108516 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:32.109149 (+ 633us) server_negotiation.cc:299] Negotiation successful
0619 14:51:32.109462 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":495,"spinlock_wait_cycles":896,"thread_start_us":325,"threads_started":1}
I20260619 14:51:32.113694 23104 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45193 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:51:32.114650 23108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:32.114354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45193 (local address 127.0.0.1:54308)
0619 14:51:32.114496 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:32.114575 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260619 14:51:32.114866 23052 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45193: connect: Connection refused (error 111)
I20260619 14:51:32.115739 23108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:32.115510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45193 (local address 127.0.0.1:54318)
0619 14:51:32.115625 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:32.115679 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260619 14:51:32.115916 23104 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:45193: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45193: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880383132139-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880383132139-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260619 14:51:32.126248 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880383132139-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880383132139-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880383132139-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880383132139-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:45193
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880383132139-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36421,127.0.98.61:45193,127.0.98.60:33439
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:51:32.165189 23104 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45193 } attempt: 1
I20260619 14:51:32.166249 23108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:32.165814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45193 (local address 127.0.0.1:54334)
0619 14:51:32.166088 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:32.166171 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156}
I20260619 14:51:32.167300 23108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:32.166950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45193 (local address 127.0.0.1:54348)
0619 14:51:32.167137 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:32.167196 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
W20260619 14:51:32.167615 23104 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:45193: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45193: connect: Connection refused (error 111)
I20260619 14:51:32.235950 23104 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45193 } attempt: 2
I20260619 14:51:32.237006 23108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:32.236689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45193 (local address 127.0.0.1:54358)
0619 14:51:32.236859 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:32.236926 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260619 14:51:32.237964 23108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:32.237735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45193 (local address 127.0.0.1:54366)
0619 14:51:32.237851 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:32.237903 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260619 14:51:32.238161 23104 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:45193: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45193: connect: Connection refused (error 111)
W20260619 14:51:32.252774 23111 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 14:51:32.253106 23111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:51:32.253131 23111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:51:32.253172 23111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:51:32.257639 23111 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:51:32.257746 23111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:51:32.257768 23111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:51:32.257786 23111 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:51:32.257802 23111 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:51:32.263787 23111 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880383132139-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880383132139-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36421,127.0.98.61:45193,127.0.98.60:33439
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880383132139-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:45193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880383132139-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23111
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880383132139-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:51:32.264993 23111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:51:32.266077 23111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:51:32.276525 23116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:51:32.276525 23119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:51:32.276583 23111 server_base.cc:1034] running on GCE node
W20260619 14:51:32.276525 23117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:51:32.277570 23111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:51:32.278936 23111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:51:32.280175 23111 hybrid_clock.cc:648] HybridClock initialized: now 1781880692280206 us; error 82 us; skew 500 ppm
Jun 19 14:51:32 dist-test-slave-2sb1 krb5kdc[23023](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880692, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:51:32.283706 23111 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260619 14:51:32.284859 23111 webserver.cc:460] Webserver started at http://127.0.98.61:44611/ using document root <none> and password file <none>
I20260619 14:51:32.285497 23111 fs_manager.cc:359] Metadata directory not provided
I20260619 14:51:32.285567 23111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:51:32.285802 23111 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:51:32.288049 23111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880383132139-392-0/minicluster-data/master-1/data/instance:
uuid: "99694446dc8c44a0a1e1a56ae06260fb"
format_stamp: "Formatted at 2026-06-19 14:51:32 on dist-test-slave-2sb1"
server_key: "8379f19662149ef5432aefb69d7d40ba"
server_key_iv: "c345214e703c07ecb71b041c7c0b9357"
server_key_version: "encryptionkey@0"
I20260619 14:51:32.288789 23111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880383132139-392-0/minicluster-data/master-1/wal/instance:
uuid: "99694446dc8c44a0a1e1a56ae06260fb"
format_stamp: "Formatted at 2026-06-19 14:51:32 on dist-test-slave-2sb1"
server_key: "8379f19662149ef5432aefb69d7d40ba"
server_key_iv: "c345214e703c07ecb71b041c7c0b9357"
server_key_version: "encryptionkey@0"
I20260619 14:51:32.293020 23111 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260619 14:51:32.296453 23126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:32.298322 23111 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260619 14:51:32.298480 23111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880383132139-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880383132139-392-0/minicluster-data/master-1/wal
uuid: "99694446dc8c44a0a1e1a56ae06260fb"
format_stamp: "Formatted at 2026-06-19 14:51:32 on dist-test-slave-2sb1"
server_key: "8379f19662149ef5432aefb69d7d40ba"
server_key_iv: "c345214e703c07ecb71b041c7c0b9357"
server_key_version: "encryptionkey@0"
I20260619 14:51:32.298691 23111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880383132139-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880383132139-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880383132139-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:51:32.320004 23111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:51:32.323901 23111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:51:32.324132 23111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:51:32.329376 23104 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45193 } attempt: 3
I20260619 14:51:32.330336 23108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:32.329969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45193 (local address 127.0.0.1:54378)
0619 14:51:32.330144 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:32.330231 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260619 14:51:32.331422 23108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:32.331077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45193 (local address 127.0.0.1:54386)
0619 14:51:32.331174 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:32.331337 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260619 14:51:32.331605 23104 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:45193: Network error: Client connection negotiation failed: client connection to 127.0.98.61:45193: connect: Connection refused (error 111)
I20260619 14:51:32.332777 23111 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:45193
I20260619 14:51:32.333066 23178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:45193 every 8 connection(s)
I20260619 14:51:32.333873 23111 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880383132139-392-0/minicluster-data/master-1/data/info.pb
I20260619 14:51:32.334667 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 23111
I20260619 14:51:32.334823 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880383132139-392-0/minicluster-data/master-1/wal/instance
I20260619 14:51:32.335158 392 external_mini_cluster.cc:1428] Setting key a953dbbc483eb4df6900c59cb7576a90
I20260619 14:51:32.337558 23179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:51:32.342113 23179 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36421 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Jun 19 14:51:32 dist-test-slave-2sb1 krb5kdc[23023](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880692, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:51:32.359946 23107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:32.344835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47534 (local address 127.0.98.62:36421)
0619 14:51:32.345015 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:32.345021 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:32.346186 (+ 1165us) server_negotiation.cc:407] Connection header received
0619 14:51:32.347688 (+ 1502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:32.347692 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:32.347756 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:32.347856 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:32.349605 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:32.350282 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:32.351848 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:32.352313 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:32.352543 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:32.355840 (+ 3297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:32.355844 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:32.355846 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:32.355877 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:32.357515 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:32.358172 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:32.358175 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:32.358176 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:32.358264 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:32.358671 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:32.358674 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:32.358676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:32.358888 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:32.359004 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:32.359617 (+ 613us) server_negotiation.cc:299] Negotiation successful
0619 14:51:32.359762 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260619 14:51:32.360975 23182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:32.345343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36421 (local address 127.0.0.1:47534)
0619 14:51:32.345953 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:32.346018 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0619 14:51:32.347388 (+ 1370us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:51:32.348184 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:51:32.348193 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:51:32.348748 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:51:32.349348 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:32.349365 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:32.350458 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:32.350462 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:32.351693 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:32.351705 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:32.352601 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:32.352606 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:32.352887 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:32.353648 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:51:32.353674 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:51:32.355581 (+ 1907us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:51:32.357675 (+ 2094us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:32.357683 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:32.357717 (+ 34us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:32.358038 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:32.358367 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:32.358370 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:32.358372 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:32.358561 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:32.359018 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:51:32.359028 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:51:32.359336 (+ 308us) client_negotiation.cc:769] Sending connection context
0619 14:51:32.359541 (+ 205us) client_negotiation.cc:240] Negotiation successful
0619 14:51:32.359941 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":151,"threads_started":1}
I20260619 14:51:32.362978 23179 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45193 } has no permanent_uuid. Determining permanent_uuid...
Jun 19 14:51:32 dist-test-slave-2sb1 krb5kdc[23023](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880692, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880383132139-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880383132139-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260619 14:51:32.374397 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
/tmp/dist-test-taskYPr0M2/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880383132139-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880383132139-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880383132139-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880383132139-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:33439
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880383132139-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36421,127.0.98.61:45193,127.0.98.60:33439
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880383132139-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880383132139-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880383132139-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260619 14:51:32.376369 23182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:32.364120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45193 (local address 127.0.0.1:54388)
0619 14:51:32.364379 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:32.364393 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:51:32.364561 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:51:32.365343 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:51:32.365346 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:51:32.365597 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:51:32.365734 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:32.365741 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:32.367038 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:32.367045 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:32.367642 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:32.367653 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:32.368706 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:32.368715 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:32.368822 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:32.369480 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:51:32.369494 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:51:32.371325 (+ 1831us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:51:32.374170 (+ 2845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:32.374176 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:32.374178 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:32.374548 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:32.375105 (+ 557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:32.375112 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:32.375117 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:32.375231 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:32.375878 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:51:32.375884 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:51:32.376001 (+ 117us) client_negotiation.cc:769] Sending connection context
0619 14:51:32.376091 (+ 90us) client_negotiation.cc:240] Negotiation successful
0619 14:51:32.376219 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173}
I20260619 14:51:32.376700 23184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:32.364329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54388 (local address 127.0.98.61:45193)
0619 14:51:32.364760 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:32.364771 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:32.364789 (+ 18us) server_negotiation.cc:407] Connection header received
0619 14:51:32.364827 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:32.364831 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:32.365101 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:32.365281 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:32.365875 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:32.366850 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:32.368000 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:32.368496 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:32.368602 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:32.371569 (+ 2967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:32.371582 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:32.371597 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:32.371626 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:32.374022 (+ 2396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:32.374716 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:32.374725 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:32.374729 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:32.374984 (+ 255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:32.375360 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:32.375363 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:32.375364 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:32.375687 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:32.375903 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:32.376230 (+ 327us) server_negotiation.cc:299] Negotiation successful
0619 14:51:32.376436 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":188,"threads_started":1}
I20260619 14:51:32.379295 23179 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33439 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:51:32.380290 23182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:32.379889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33439 (local address 127.0.0.1:46374)
0619 14:51:32.380066 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:32.380176 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260619 14:51:32.380584 23129 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33439: connect: Connection refused (error 111)
I20260619 14:51:32.381711 23182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:32.381393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33439 (local address 127.0.0.1:46378)
0619 14:51:32.381547 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:32.381615 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260619 14:51:32.382058 23179 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33439: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33439: connect: Connection refused (error 111)
I20260619 14:51:32.431499 23179 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33439 } attempt: 1
I20260619 14:51:32.432574 23182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:32.432215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33439 (local address 127.0.0.1:46390)
0619 14:51:32.432398 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:32.432476 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260619 14:51:32.433547 23182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:32.433290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33439 (local address 127.0.0.1:46398)
0619 14:51:32.433412 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:32.433475 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260619 14:51:32.433869 23179 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33439: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33439: connect: Connection refused (error 111)
I20260619 14:51:32.474808 23104 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45193 } attempt: 4
Jun 19 14:51:32 dist-test-slave-2sb1 krb5kdc[23023](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880692, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 14:51:32.485872 23184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:32.475639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54400 (local address 127.0.98.61:45193)
0619 14:51:32.475761 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:32.475766 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:32.475931 (+ 165us) server_negotiation.cc:407] Connection header received
0619 14:51:32.475976 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:32.475978 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:32.476023 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:32.476117 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:32.476582 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:32.477228 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:32.478091 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:32.478903 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:32.479041 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:32.481807 (+ 2766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:32.481810 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:32.481813 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:32.481842 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:32.483582 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:32.484153 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:32.484157 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:32.484159 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:32.484252 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:32.484826 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:32.484830 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:32.484832 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:32.485099 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:32.485239 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:32.485551 (+ 312us) server_negotiation.cc:299] Negotiation successful
0619 14:51:32.485689 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260619 14:51:32.486027 23108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:32.475467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45193 (local address 127.0.0.1:54400)
0619 14:51:32.475645 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:32.475661 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:51:32.475766 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:51:32.476137 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:51:32.476140 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:51:32.476347 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:51:32.476455 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:32.476460 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:32.477368 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:32.477371 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:32.477953 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:32.477962 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:32.479049 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:32.479053 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:32.479171 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:32.480007 (+ 836us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:51:32.480022 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:51:32.481619 (+ 1597us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:51:32.483770 (+ 2151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:32.483774 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:32.483777 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:32.484032 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:32.484357 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:32.484362 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:32.484364 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:32.484584 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:32.485226 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:51:32.485230 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:51:32.485351 (+ 121us) client_negotiation.cc:769] Sending connection context
0619 14:51:32.485489 (+ 138us) client_negotiation.cc:240] Negotiation successful
0619 14:51:32.485675 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85}
I20260619 14:51:32.487218 23104 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33439 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:51:32.488091 23108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:32.487770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33439 (local address 127.0.0.1:46400)
0619 14:51:32.487924 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:32.487990 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260619 14:51:32.489166 23108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:32.488839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33439 (local address 127.0.0.1:46414)
0619 14:51:32.489061 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:32.489107 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161}
W20260619 14:51:32.489363 23104 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33439: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33439: connect: Connection refused (error 111)
I20260619 14:51:32.502091 23179 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33439 } attempt: 2
I20260619 14:51:32.503170 23182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:32.502750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33439 (local address 127.0.0.1:46418)
0619 14:51:32.502928 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:32.503039 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260619 14:51:32.504097 23182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:32.503837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33439 (local address 127.0.0.1:46434)
0619 14:51:32.503965 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:32.504028 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260619 14:51:32.504253 23179 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33439: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33439: connect: Connection refused (error 111)
W20260619 14:51:32.510304 23185 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260619 14:51:32.510907 23185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260619 14:51:32.511027 23185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260619 14:51:32.511139 23185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260619 14:51:32.515724 23185 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260619 14:51:32.515867 23185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260619 14:51:32.515908 23185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260619 14:51:32.515946 23185 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260619 14:51:32.515985 23185 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260619 14:51:32.521908 23185 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880383132139-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880383132139-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36421,127.0.98.61:45193,127.0.98.60:33439
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880383132139-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:33439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880383132139-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23185
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880383132139-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 19 Jun 2026 14:11:05 UTC on ccycloud.quasar-fhnmjr.root.comops.site
build id 4455769
I20260619 14:51:32.523113 23185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260619 14:51:32.524201 23185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260619 14:51:32.533670 23192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:51:32.533663 23194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260619 14:51:32.533669 23191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260619 14:51:32.534029 23185 server_base.cc:1034] running on GCE node
I20260619 14:51:32.534790 23185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260619 14:51:32.536095 23185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260619 14:51:32.537331 23185 hybrid_clock.cc:648] HybridClock initialized: now 1781880692537312 us; error 42 us; skew 500 ppm
Jun 19 14:51:32 dist-test-slave-2sb1 krb5kdc[23023](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880692, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260619 14:51:32.540907 23185 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260619 14:51:32.542222 23185 webserver.cc:460] Webserver started at http://127.0.98.60:41623/ using document root <none> and password file <none>
I20260619 14:51:32.542894 23185 fs_manager.cc:359] Metadata directory not provided
I20260619 14:51:32.543008 23185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260619 14:51:32.543331 23185 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260619 14:51:32.545897 23185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880383132139-392-0/minicluster-data/master-2/data/instance:
uuid: "ba35da5abc154f0e84d358b48a167607"
format_stamp: "Formatted at 2026-06-19 14:51:32 on dist-test-slave-2sb1"
server_key: "d66899fb973acf0ee77ad37290b44bb8"
server_key_iv: "5eff9d5d3812419c45ba5c1817c34086"
server_key_version: "encryptionkey@0"
I20260619 14:51:32.546656 23185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880383132139-392-0/minicluster-data/master-2/wal/instance:
uuid: "ba35da5abc154f0e84d358b48a167607"
format_stamp: "Formatted at 2026-06-19 14:51:32 on dist-test-slave-2sb1"
server_key: "d66899fb973acf0ee77ad37290b44bb8"
server_key_iv: "5eff9d5d3812419c45ba5c1817c34086"
server_key_version: "encryptionkey@0"
I20260619 14:51:32.548579 23104 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33439 } attempt: 1
I20260619 14:51:32.549609 23108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:32.549175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33439 (local address 127.0.0.1:46450)
0619 14:51:32.549377 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:32.549524 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260619 14:51:32.550611 23108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:32.550334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33439 (local address 127.0.0.1:46458)
0619 14:51:32.550472 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:32.550534 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260619 14:51:32.550911 23104 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33439: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33439: connect: Connection refused (error 111)
I20260619 14:51:32.551789 23185 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260619 14:51:32.555038 23201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260619 14:51:32.556816 23185 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260619 14:51:32.556994 23185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880383132139-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880383132139-392-0/minicluster-data/master-2/wal
uuid: "ba35da5abc154f0e84d358b48a167607"
format_stamp: "Formatted at 2026-06-19 14:51:32 on dist-test-slave-2sb1"
server_key: "d66899fb973acf0ee77ad37290b44bb8"
server_key_iv: "5eff9d5d3812419c45ba5c1817c34086"
server_key_version: "encryptionkey@0"
I20260619 14:51:32.557140 23185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880383132139-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880383132139-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880383132139-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260619 14:51:32.572062 23185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260619 14:51:32.576136 23185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260619 14:51:32.576325 23185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260619 14:51:32.584900 23185 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:33439
I20260619 14:51:32.585211 23253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:33439 every 8 connection(s)
I20260619 14:51:32.586144 23185 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880383132139-392-0/minicluster-data/master-2/data/info.pb
I20260619 14:51:32.589329 23254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260619 14:51:32.593457 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu as pid 23185
I20260619 14:51:32.593611 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskYPr0M2/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781880383132139-392-0/minicluster-data/master-2/wal/instance
I20260619 14:51:32.593933 392 external_mini_cluster.cc:1428] Setting key fc42b3d1bd10e524cd50f958ba9e6192
I20260619 14:51:32.593683 23254 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36421 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:51:32.595439 23179 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33439 } attempt: 3
Jun 19 14:51:32 dist-test-slave-2sb1 krb5kdc[23023](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880692, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 19 14:51:32 dist-test-slave-2sb1 krb5kdc[23023](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880692, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Jun 19 14:51:32 dist-test-slave-2sb1 krb5kdc[23023](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880691, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 14:51:32.609790 23182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:32.595939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33439 (local address 127.0.0.1:46472)
0619 14:51:32.596151 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:32.596168 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 14:51:32.596253 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:51:32.598078 (+ 1825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:51:32.598085 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:51:32.598349 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:51:32.598483 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:32.598494 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:32.600359 (+ 1865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:32.600362 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:32.600984 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:32.600995 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:32.601800 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:32.601806 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:32.601923 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:32.602679 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:51:32.602749 (+ 70us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:51:32.604597 (+ 1848us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:51:32.607503 (+ 2906us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:32.607509 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:32.607514 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:32.607821 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:32.608232 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:32.608235 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:32.608237 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:32.608337 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:32.609187 (+ 850us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:51:32.609191 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:51:32.609298 (+ 107us) client_negotiation.cc:769] Sending connection context
0619 14:51:32.609432 (+ 134us) client_negotiation.cc:240] Negotiation successful
0619 14:51:32.609603 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116}
I20260619 14:51:32.609859 23107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:32.596356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47538 (local address 127.0.98.62:36421)
0619 14:51:32.596484 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:32.596487 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:32.596911 (+ 424us) server_negotiation.cc:407] Connection header received
0619 14:51:32.597931 (+ 1020us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:32.597934 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:32.598028 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:32.598123 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:32.599157 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:32.599800 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:32.601490 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:32.601833 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:32.601928 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:32.605668 (+ 3740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:32.605673 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:32.605676 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:32.605711 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:32.607371 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:32.608105 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:32.608109 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:32.608113 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:32.608221 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:32.608682 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:32.608688 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:32.608690 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:32.608967 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:32.609036 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:32.609571 (+ 535us) server_negotiation.cc:299] Negotiation successful
0619 14:51:32.609707 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260619 14:51:32.610984 23259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:32.597064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46472 (local address 127.0.98.60:33439)
0619 14:51:32.597321 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:32.597329 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:32.597342 (+ 13us) server_negotiation.cc:407] Connection header received
0619 14:51:32.597518 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:32.597545 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:32.597868 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:32.598065 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:32.598600 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:32.600176 (+ 1576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:32.601133 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:32.601657 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:32.601831 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:32.604797 (+ 2966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:32.604803 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:32.604816 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:32.604841 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:32.607330 (+ 2489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:32.607939 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:32.607948 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:32.607953 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:32.608104 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:32.608452 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:32.608456 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:32.608458 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:32.609061 (+ 603us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:32.609208 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:32.609553 (+ 345us) server_negotiation.cc:299] Negotiation successful
0619 14:51:32.609892 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":185,"spinlock_wait_cycles":11136,"thread_start_us":85,"threads_started":1}
I20260619 14:51:32.610980 23258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:32.596271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36421 (local address 127.0.0.1:47538)
0619 14:51:32.596638 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:32.596717 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0619 14:51:32.597532 (+ 815us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:51:32.598322 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:51:32.598330 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:51:32.598829 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:51:32.598998 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:32.599008 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:32.599964 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:32.599968 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:32.601341 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:32.601352 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:32.602080 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:32.602082 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:32.602157 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:32.602990 (+ 833us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:51:32.603011 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:51:32.605486 (+ 2475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:51:32.607545 (+ 2059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:32.607556 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:32.607571 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:32.607936 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:32.608371 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:32.608374 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:32.608376 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:32.608523 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:32.609099 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:51:32.609104 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:51:32.609298 (+ 194us) client_negotiation.cc:769] Sending connection context
0619 14:51:32.610044 (+ 746us) client_negotiation.cc:240] Negotiation successful
0619 14:51:32.610230 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"spinlock_wait_cycles":2432,"thread_start_us":95,"threads_started":1}
I20260619 14:51:32.611671 23257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:32.595454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46468 (local address 127.0.98.60:33439)
0619 14:51:32.596193 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:32.596217 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:32.596261 (+ 44us) server_negotiation.cc:407] Connection header received
0619 14:51:32.597518 (+ 1257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:32.597545 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:32.597868 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:32.598065 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:32.599166 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:32.600176 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:32.601558 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:32.601871 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:32.602380 (+ 509us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:32.606166 (+ 3786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:32.606171 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:32.606174 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:32.606203 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:32.608809 (+ 2606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:32.609306 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:32.609311 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:32.609316 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:32.609432 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:32.609767 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:32.609770 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:32.609772 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:32.610451 (+ 679us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:32.610626 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:32.610883 (+ 257us) server_negotiation.cc:299] Negotiation successful
0619 14:51:32.610999 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":724,"server-negotiator.queue_time_us":414,"spinlock_wait_cycles":9600,"thread_start_us":196,"threads_started":1}
I20260619 14:51:32.613268 23254 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45193 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:51:32.616636 23179 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99694446dc8c44a0a1e1a56ae06260fb: Bootstrap starting.
I20260619 14:51:32.618105 23104 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33439 } attempt: 2
Jun 19 14:51:32 dist-test-slave-2sb1 krb5kdc[23023](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880692, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 14:51:32.620944 23179 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 99694446dc8c44a0a1e1a56ae06260fb: Neither blocks nor log segments found. Creating new log.
Jun 19 14:51:32 dist-test-slave-2sb1 krb5kdc[23023](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880691, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260619 14:51:32.622382 23179 log.cc:824] T 00000000000000000000000000000000 P 99694446dc8c44a0a1e1a56ae06260fb: Log is configured to *not* fsync() on all Append() calls
I20260619 14:51:32.624712 23184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:32.614457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54410 (local address 127.0.98.61:45193)
0619 14:51:32.614578 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:32.614582 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:32.614594 (+ 12us) server_negotiation.cc:407] Connection header received
0619 14:51:32.614631 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:32.614634 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:32.614683 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:32.614761 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:32.615268 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:32.615925 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:32.616964 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:32.617224 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:32.617343 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:32.620159 (+ 2816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:32.620164 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:32.620166 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:32.620198 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:32.622160 (+ 1962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:32.622948 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:32.622954 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:32.622977 (+ 23us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:32.623077 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:32.623474 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:32.623479 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:32.623482 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:32.623802 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:32.623887 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:32.624329 (+ 442us) server_negotiation.cc:299] Negotiation successful
0619 14:51:32.624481 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260619 14:51:32.625590 23179 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99694446dc8c44a0a1e1a56ae06260fb: No bootstrap required, opened a new log
I20260619 14:51:32.625790 23258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:32.613802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:45193 (local address 127.0.0.1:54410)
0619 14:51:32.613920 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:32.613937 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0619 14:51:32.614019 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:51:32.614886 (+ 867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:51:32.614888 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:51:32.615037 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:51:32.615135 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:32.615140 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:32.616056 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:32.616059 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:32.616838 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:32.616851 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:32.617345 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:32.617349 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:32.617455 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:32.618061 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:51:32.618077 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:51:32.619991 (+ 1914us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:51:32.622388 (+ 2397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:32.622392 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:32.622394 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:32.622770 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:32.623220 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:32.623224 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:32.623227 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:32.623346 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:32.623971 (+ 625us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:51:32.623974 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:51:32.624085 (+ 111us) client_negotiation.cc:769] Sending connection context
0619 14:51:32.625331 (+ 1246us) client_negotiation.cc:240] Negotiation successful
0619 14:51:32.625514 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
Jun 19 14:51:32 dist-test-slave-2sb1 krb5kdc[23023](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880692, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 14:51:32.627019 23254 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33439 } has no permanent_uuid. Determining permanent_uuid...
I20260619 14:51:32.627341 23107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:32.614396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47540 (local address 127.0.98.62:36421)
0619 14:51:32.614509 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:32.614514 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:32.614528 (+ 14us) server_negotiation.cc:407] Connection header received
0619 14:51:32.614567 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:32.614570 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:32.614618 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:32.614698 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:32.615641 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:32.616348 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:32.617510 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:32.617881 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:32.618008 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:32.622380 (+ 4372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:32.622384 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:32.622387 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:32.622416 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:32.624342 (+ 1926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:32.625233 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:32.625239 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:32.625241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:32.625339 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:32.625713 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:32.625717 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:32.625718 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:32.625995 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:32.626254 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:32.626687 (+ 433us) server_negotiation.cc:299] Negotiation successful
0619 14:51:32.626842 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260619 14:51:32.632166 23108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:32.618556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33439 (local address 127.0.0.1:46484)
0619 14:51:32.618864 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:32.618880 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0619 14:51:32.618960 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:51:32.620544 (+ 1584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:51:32.620548 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:51:32.620812 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:51:32.620942 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:32.620951 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:32.622182 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:32.622189 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:32.623389 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:32.623404 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:32.624122 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:32.624126 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:32.624218 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:32.624744 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:51:32.624761 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:51:32.626422 (+ 1661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:51:32.629581 (+ 3159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:32.629587 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:32.629589 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:32.629885 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:32.630239 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:32.630242 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:32.630245 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:32.630480 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:32.631159 (+ 679us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:51:32.631163 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:51:32.631638 (+ 475us) client_negotiation.cc:769] Sending connection context
0619 14:51:32.631779 (+ 141us) client_negotiation.cc:240] Negotiation successful
0619 14:51:32.631967 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217}
I20260619 14:51:32.631969 23179 raft_consensus.cc:348] T 00000000000000000000000000000000 P 99694446dc8c44a0a1e1a56ae06260fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7e69f99b4a94fcb8376d048395019ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36421 } } peers { permanent_uuid: "99694446dc8c44a0a1e1a56ae06260fb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45193 } } peers { permanent_uuid: "ba35da5abc154f0e84d358b48a167607" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33439 } }
I20260619 14:51:32.632274 23179 raft_consensus.cc:729] T 00000000000000000000000000000000 P 99694446dc8c44a0a1e1a56ae06260fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99694446dc8c44a0a1e1a56ae06260fb, State: Initialized, Role: FOLLOWER
I20260619 14:51:32.633004 23179 consensus_queue.cc:260] T 00000000000000000000000000000000 P 99694446dc8c44a0a1e1a56ae06260fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7e69f99b4a94fcb8376d048395019ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36421 } } peers { permanent_uuid: "99694446dc8c44a0a1e1a56ae06260fb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45193 } } peers { permanent_uuid: "ba35da5abc154f0e84d358b48a167607" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33439 } }
I20260619 14:51:32.633138 23257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:32.618880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46484 (local address 127.0.98.60:33439)
0619 14:51:32.620272 (+ 1392us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:32.620278 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:32.620292 (+ 14us) server_negotiation.cc:407] Connection header received
0619 14:51:32.620333 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:32.620335 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:32.620394 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:32.621078 (+ 684us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:32.621183 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:32.622019 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:32.623582 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:32.623963 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:32.624467 (+ 504us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:32.626621 (+ 2154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:32.626625 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:32.626628 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:32.626656 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:32.629432 (+ 2776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:32.630026 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:32.630030 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:32.630032 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:32.630120 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:32.630641 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:32.630645 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:32.630647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:32.631001 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:32.632560 (+ 1559us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:32.632701 (+ 141us) server_negotiation.cc:299] Negotiation successful
0619 14:51:32.632955 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1307,"spinlock_wait_cycles":77568}
Jun 19 14:51:32 dist-test-slave-2sb1 krb5kdc[23023](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880692, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260619 14:51:32.635495 23179 sys_catalog.cc:564] T 00000000000000000000000000000000 P 99694446dc8c44a0a1e1a56ae06260fb [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:51:32.637539 23104 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7e69f99b4a94fcb8376d048395019ba: Bootstrap starting.
Jun 19 14:51:32 dist-test-slave-2sb1 krb5kdc[23023](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781880691, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260619 14:51:32.640229 23258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:32.627541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33439 (local address 127.0.0.1:46498)
0619 14:51:32.628015 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0619 14:51:32.628029 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0619 14:51:32.628133 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0619 14:51:32.628500 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0619 14:51:32.628503 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0619 14:51:32.628903 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0619 14:51:32.629104 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:32.629116 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:32.630220 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:32.630223 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:32.631227 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0619 14:51:32.631238 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0619 14:51:32.632172 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0619 14:51:32.632176 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0619 14:51:32.632602 (+ 426us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:32.633596 (+ 994us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0619 14:51:32.633612 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0619 14:51:32.636062 (+ 2450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0619 14:51:32.638295 (+ 2233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:32.638302 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:32.638306 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:32.638597 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:32.639020 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0619 14:51:32.639023 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0619 14:51:32.639025 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0619 14:51:32.639113 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0619 14:51:32.639604 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0619 14:51:32.639610 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0619 14:51:32.639695 (+ 85us) client_negotiation.cc:769] Sending connection context
0619 14:51:32.639842 (+ 147us) client_negotiation.cc:240] Negotiation successful
0619 14:51:32.640009 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425}
I20260619 14:51:32.641321 23104 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a7e69f99b4a94fcb8376d048395019ba: Neither blocks nor log segments found. Creating new log.
I20260619 14:51:32.641993 23259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:32.627788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46498 (local address 127.0.98.60:33439)
0619 14:51:32.627914 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:32.627918 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:32.628077 (+ 159us) server_negotiation.cc:407] Connection header received
0619 14:51:32.628232 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:32.628237 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:32.628297 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:32.628347 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:32.629251 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:32.630091 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:32.631358 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:32.631757 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:32.632023 (+ 266us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:32.636238 (+ 4215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:32.636243 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:32.636248 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:32.636277 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:32.638165 (+ 1888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:32.638776 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:32.638780 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:32.638784 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:32.638912 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:32.639214 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:32.639220 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:32.639223 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:32.639464 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:32.639566 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:32.641261 (+ 1695us) server_negotiation.cc:299] Negotiation successful
0619 14:51:32.641417 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 14:51:32.642383 23104 log.cc:824] T 00000000000000000000000000000000 P a7e69f99b4a94fcb8376d048395019ba: Log is configured to *not* fsync() on all Append() calls
I20260619 14:51:32.644627 23184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:32.629764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54416 (local address 127.0.98.61:45193)
0619 14:51:32.630019 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:32.630023 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:32.630189 (+ 166us) server_negotiation.cc:407] Connection header received
0619 14:51:32.630532 (+ 343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:32.630536 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:32.630592 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:32.630645 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:32.631636 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:32.632681 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:32.635090 (+ 2409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:32.635433 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:32.635518 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:32.640511 (+ 4993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:32.640515 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:32.640518 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:32.640548 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:32.642393 (+ 1845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:32.643190 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:32.643195 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:32.643197 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:32.643296 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:32.643660 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:32.643663 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:32.643665 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:32.643888 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:32.644102 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:32.644310 (+ 208us) server_negotiation.cc:299] Negotiation successful
0619 14:51:32.644442 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179}
I20260619 14:51:32.640280 23111 master_runner.cc:186] Error getting master registration for 127.0.98.62:36421: OK, instance_id { permanent_uuid: "a7e69f99b4a94fcb8376d048395019ba" instance_seqno: 1781880692078463 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260619 14:51:32.645125 23111 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 14:51:32.645591 23261 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99694446dc8c44a0a1e1a56ae06260fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7e69f99b4a94fcb8376d048395019ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36421 } } peers { permanent_uuid: "99694446dc8c44a0a1e1a56ae06260fb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45193 } } peers { permanent_uuid: "ba35da5abc154f0e84d358b48a167607" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33439 } } }
I20260619 14:51:32.645746 23261 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99694446dc8c44a0a1e1a56ae06260fb [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:51:32.645813 23254 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba35da5abc154f0e84d358b48a167607: Bootstrap starting.
I20260619 14:51:32.648545 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260619 14:51:32.648962 23254 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba35da5abc154f0e84d358b48a167607: Neither blocks nor log segments found. Creating new log.
W20260619 14:51:32.649163 23272 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 99694446dc8c44a0a1e1a56ae06260fb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:51:32.649247 23272 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:51:32.650008 23254 log.cc:824] T 00000000000000000000000000000000 P ba35da5abc154f0e84d358b48a167607: Log is configured to *not* fsync() on all Append() calls
I20260619 14:51:32.649993 23078 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "a7e69f99b4a94fcb8376d048395019ba"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47540
I20260619 14:51:32.652817 23254 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba35da5abc154f0e84d358b48a167607: No bootstrap required, opened a new log
I20260619 14:51:32.653913 23104 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7e69f99b4a94fcb8376d048395019ba: No bootstrap required, opened a new log
I20260619 14:51:32.657459 23104 raft_consensus.cc:348] T 00000000000000000000000000000000 P a7e69f99b4a94fcb8376d048395019ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7e69f99b4a94fcb8376d048395019ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36421 } } peers { permanent_uuid: "99694446dc8c44a0a1e1a56ae06260fb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45193 } } peers { permanent_uuid: "ba35da5abc154f0e84d358b48a167607" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33439 } }
I20260619 14:51:32.657752 23104 raft_consensus.cc:729] T 00000000000000000000000000000000 P a7e69f99b4a94fcb8376d048395019ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7e69f99b4a94fcb8376d048395019ba, State: Initialized, Role: FOLLOWER
I20260619 14:51:32.657565 23254 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba35da5abc154f0e84d358b48a167607 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7e69f99b4a94fcb8376d048395019ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36421 } } peers { permanent_uuid: "99694446dc8c44a0a1e1a56ae06260fb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45193 } } peers { permanent_uuid: "ba35da5abc154f0e84d358b48a167607" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33439 } }
I20260619 14:51:32.657859 23254 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba35da5abc154f0e84d358b48a167607 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba35da5abc154f0e84d358b48a167607, State: Initialized, Role: FOLLOWER
I20260619 14:51:32.658129 23104 consensus_queue.cc:260] T 00000000000000000000000000000000 P a7e69f99b4a94fcb8376d048395019ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7e69f99b4a94fcb8376d048395019ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36421 } } peers { permanent_uuid: "99694446dc8c44a0a1e1a56ae06260fb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45193 } } peers { permanent_uuid: "ba35da5abc154f0e84d358b48a167607" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33439 } }
I20260619 14:51:32.658260 23254 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba35da5abc154f0e84d358b48a167607 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7e69f99b4a94fcb8376d048395019ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36421 } } peers { permanent_uuid: "99694446dc8c44a0a1e1a56ae06260fb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45193 } } peers { permanent_uuid: "ba35da5abc154f0e84d358b48a167607" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33439 } }
I20260619 14:51:32.658805 23276 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7e69f99b4a94fcb8376d048395019ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7e69f99b4a94fcb8376d048395019ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36421 } } peers { permanent_uuid: "99694446dc8c44a0a1e1a56ae06260fb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45193 } } peers { permanent_uuid: "ba35da5abc154f0e84d358b48a167607" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33439 } } }
I20260619 14:51:32.659243 23276 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7e69f99b4a94fcb8376d048395019ba [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:51:32.659642 23104 sys_catalog.cc:564] T 00000000000000000000000000000000 P a7e69f99b4a94fcb8376d048395019ba [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:51:32.659708 23078 raft_consensus.cc:482] T 00000000000000000000000000000000 P a7e69f99b4a94fcb8376d048395019ba [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260619 14:51:32.659730 23254 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba35da5abc154f0e84d358b48a167607 [sys.catalog]: configured and running, proceeding with master startup.
I20260619 14:51:32.659833 23078 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a7e69f99b4a94fcb8376d048395019ba [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:51:32.661641 23078 raft_consensus.cc:504] T 00000000000000000000000000000000 P a7e69f99b4a94fcb8376d048395019ba [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7e69f99b4a94fcb8376d048395019ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36421 } } peers { permanent_uuid: "99694446dc8c44a0a1e1a56ae06260fb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45193 } } peers { permanent_uuid: "ba35da5abc154f0e84d358b48a167607" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33439 } }
I20260619 14:51:32.662600 23277 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba35da5abc154f0e84d358b48a167607 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7e69f99b4a94fcb8376d048395019ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36421 } } peers { permanent_uuid: "99694446dc8c44a0a1e1a56ae06260fb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45193 } } peers { permanent_uuid: "ba35da5abc154f0e84d358b48a167607" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33439 } } }
I20260619 14:51:32.662791 23277 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba35da5abc154f0e84d358b48a167607 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:51:32.663887 23153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a7e69f99b4a94fcb8376d048395019ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "99694446dc8c44a0a1e1a56ae06260fb"
I20260619 14:51:32.664186 23153 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 99694446dc8c44a0a1e1a56ae06260fb [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:51:32.665846 23153 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 99694446dc8c44a0a1e1a56ae06260fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7e69f99b4a94fcb8376d048395019ba in term 1.
W20260619 14:51:32.666960 23299 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ba35da5abc154f0e84d358b48a167607: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:51:32.666958 23295 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a7e69f99b4a94fcb8376d048395019ba: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260619 14:51:32.667050 23299 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260619 14:51:32.667096 23295 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260619 14:51:32.668757 23052 leader_election.cc:302] T 00000000000000000000000000000000 P a7e69f99b4a94fcb8376d048395019ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99694446dc8c44a0a1e1a56ae06260fb, a7e69f99b4a94fcb8376d048395019ba; no voters:
I20260619 14:51:32.669500 23078 leader_election.cc:288] T 00000000000000000000000000000000 P a7e69f99b4a94fcb8376d048395019ba [CANDIDATE]: Term 1 election: Requested vote from peers 99694446dc8c44a0a1e1a56ae06260fb (127.0.98.61:45193), ba35da5abc154f0e84d358b48a167607 (127.0.98.60:33439)
I20260619 14:51:32.673592 23228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a7e69f99b4a94fcb8376d048395019ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "ba35da5abc154f0e84d358b48a167607"
I20260619 14:51:32.673936 23228 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba35da5abc154f0e84d358b48a167607 [term 0 FOLLOWER]: Advancing to term 1
I20260619 14:51:32.675557 23228 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ba35da5abc154f0e84d358b48a167607 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7e69f99b4a94fcb8376d048395019ba in term 1.
I20260619 14:51:32.681679 23185 master_runner.cc:263] No leader master found from master ba35da5abc154f0e84d358b48a167607
I20260619 14:51:32.681784 23185 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 14:51:32.685099 23036 master_runner.cc:263] No leader master found from master a7e69f99b4a94fcb8376d048395019ba
I20260619 14:51:32.685227 23036 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260619 14:51:32.685428 23276 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a7e69f99b4a94fcb8376d048395019ba [term 1 FOLLOWER]: Leader election won for term 1
I20260619 14:51:32.685792 23276 raft_consensus.cc:686] T 00000000000000000000000000000000 P a7e69f99b4a94fcb8376d048395019ba [term 1 LEADER]: Becoming Leader. State: Replica: a7e69f99b4a94fcb8376d048395019ba, State: Running, Role: LEADER
I20260619 14:51:32.686272 23276 consensus_queue.cc:237] T 00000000000000000000000000000000 P a7e69f99b4a94fcb8376d048395019ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7e69f99b4a94fcb8376d048395019ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36421 } } peers { permanent_uuid: "99694446dc8c44a0a1e1a56ae06260fb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45193 } } peers { permanent_uuid: "ba35da5abc154f0e84d358b48a167607" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33439 } }
I20260619 14:51:32.688872 23276 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7e69f99b4a94fcb8376d048395019ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader a7e69f99b4a94fcb8376d048395019ba. Latest consensus state: current_term: 1 leader_uuid: "a7e69f99b4a94fcb8376d048395019ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7e69f99b4a94fcb8376d048395019ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36421 } } peers { permanent_uuid: "99694446dc8c44a0a1e1a56ae06260fb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45193 } } peers { permanent_uuid: "ba35da5abc154f0e84d358b48a167607" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33439 } } }
I20260619 14:51:32.689038 23276 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7e69f99b4a94fcb8376d048395019ba [sys.catalog]: This master's current role is: LEADER
I20260619 14:51:32.691691 23308 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260619 14:51:32.695242 23308 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260619 14:51:32.709985 23107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:32.689825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47550 (local address 127.0.98.62:36421)
0619 14:51:32.689962 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:32.689967 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:32.697974 (+ 8007us) server_negotiation.cc:407] Connection header received
0619 14:51:32.698193 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:32.698196 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:32.698248 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:32.698313 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:32.699381 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:32.700176 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:32.701383 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:32.701747 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:32.701844 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:32.704177 (+ 2333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:32.704181 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:32.704184 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:32.704221 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:32.707519 (+ 3298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:32.707993 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:32.707996 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:32.707998 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:32.708084 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:32.708398 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:32.708402 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:32.708403 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:32.708702 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:32.708824 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:32.709441 (+ 617us) server_negotiation.cc:299] Negotiation successful
0619 14:51:32.709651 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260619 14:51:32.713428 23184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:32.690655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54420 (local address 127.0.98.61:45193)
0619 14:51:32.690789 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:32.690794 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:32.700940 (+ 10146us) server_negotiation.cc:407] Connection header received
0619 14:51:32.701113 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:32.701117 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:32.701170 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:32.701222 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:32.702229 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:32.703122 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:32.704694 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:32.705028 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:32.705131 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:32.708406 (+ 3275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:32.708410 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:32.708413 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:32.708448 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:32.711383 (+ 2935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:32.711878 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:32.711885 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:32.711889 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:32.711989 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:32.712312 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:32.712316 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:32.712318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:32.712537 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:32.712691 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:32.713045 (+ 354us) server_negotiation.cc:299] Negotiation successful
0619 14:51:32.713165 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260619 14:51:32.718598 23228 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ba35da5abc154f0e84d358b48a167607 [term 1 FOLLOWER]: Refusing update from remote peer a7e69f99b4a94fcb8376d048395019ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:51:32.720311 23306 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a7e69f99b4a94fcb8376d048395019ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba35da5abc154f0e84d358b48a167607" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:51:32.722463 23259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:32.693082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46504 (local address 127.0.98.60:33439)
0619 14:51:32.693211 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:32.693215 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:32.704564 (+ 11349us) server_negotiation.cc:407] Connection header received
0619 14:51:32.704839 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:32.704842 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:32.704897 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:32.704948 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:32.705872 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:32.706626 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:32.708806 (+ 2180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:32.710522 (+ 1716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:32.710611 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:32.716216 (+ 5605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0619 14:51:32.716221 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0619 14:51:32.716224 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0619 14:51:32.716258 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0619 14:51:32.720085 (+ 3827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:32.720706 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:32.720710 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:32.720712 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:32.720825 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0619 14:51:32.721141 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0619 14:51:32.721145 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0619 14:51:32.721146 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0619 14:51:32.721381 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0619 14:51:32.721506 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:32.722194 (+ 688us) server_negotiation.cc:299] Negotiation successful
0619 14:51:32.722314 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260619 14:51:32.725523 23277 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba35da5abc154f0e84d358b48a167607 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a7e69f99b4a94fcb8376d048395019ba. Latest consensus state: current_term: 1 leader_uuid: "a7e69f99b4a94fcb8376d048395019ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7e69f99b4a94fcb8376d048395019ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36421 } } peers { permanent_uuid: "99694446dc8c44a0a1e1a56ae06260fb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45193 } } peers { permanent_uuid: "ba35da5abc154f0e84d358b48a167607" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33439 } } }
I20260619 14:51:32.725709 23277 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba35da5abc154f0e84d358b48a167607 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:51:32.730055 23153 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 99694446dc8c44a0a1e1a56ae06260fb [term 1 FOLLOWER]: Refusing update from remote peer a7e69f99b4a94fcb8376d048395019ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260619 14:51:32.733163 23306 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a7e69f99b4a94fcb8376d048395019ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "99694446dc8c44a0a1e1a56ae06260fb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260619 14:51:32.738909 23261 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99694446dc8c44a0a1e1a56ae06260fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader a7e69f99b4a94fcb8376d048395019ba. Latest consensus state: current_term: 1 leader_uuid: "a7e69f99b4a94fcb8376d048395019ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7e69f99b4a94fcb8376d048395019ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36421 } } peers { permanent_uuid: "99694446dc8c44a0a1e1a56ae06260fb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45193 } } peers { permanent_uuid: "ba35da5abc154f0e84d358b48a167607" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33439 } } }
I20260619 14:51:32.739084 23261 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99694446dc8c44a0a1e1a56ae06260fb [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:51:32.747601 23306 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7e69f99b4a94fcb8376d048395019ba [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a7e69f99b4a94fcb8376d048395019ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7e69f99b4a94fcb8376d048395019ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36421 } } peers { permanent_uuid: "99694446dc8c44a0a1e1a56ae06260fb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45193 } } peers { permanent_uuid: "ba35da5abc154f0e84d358b48a167607" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33439 } } }
I20260619 14:51:32.747782 23306 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7e69f99b4a94fcb8376d048395019ba [sys.catalog]: This master's current role is: LEADER
I20260619 14:51:32.750732 23261 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99694446dc8c44a0a1e1a56ae06260fb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a7e69f99b4a94fcb8376d048395019ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7e69f99b4a94fcb8376d048395019ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36421 } } peers { permanent_uuid: "99694446dc8c44a0a1e1a56ae06260fb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45193 } } peers { permanent_uuid: "ba35da5abc154f0e84d358b48a167607" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33439 } } }
I20260619 14:51:32.750874 23261 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99694446dc8c44a0a1e1a56ae06260fb [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:51:32.753039 23312 mvcc.cc:205] Tried to move back new op lower bound from 7298583317357260800 to 7298583317248483328. Current Snapshot: MvccSnapshot[applied={T|T < 7298583317357260800}]
I20260619 14:51:32.754587 23315 mvcc.cc:205] Tried to move back new op lower bound from 7298583317357260800 to 7298583317248483328. Current Snapshot: MvccSnapshot[applied={T|T < 7298583317357260800}]
I20260619 14:51:32.755517 23276 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7e69f99b4a94fcb8376d048395019ba [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a7e69f99b4a94fcb8376d048395019ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7e69f99b4a94fcb8376d048395019ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36421 } } peers { permanent_uuid: "99694446dc8c44a0a1e1a56ae06260fb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45193 } } peers { permanent_uuid: "ba35da5abc154f0e84d358b48a167607" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33439 } } }
I20260619 14:51:32.755679 23276 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7e69f99b4a94fcb8376d048395019ba [sys.catalog]: This master's current role is: LEADER
I20260619 14:51:32.757100 23277 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba35da5abc154f0e84d358b48a167607 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a7e69f99b4a94fcb8376d048395019ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7e69f99b4a94fcb8376d048395019ba" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36421 } } peers { permanent_uuid: "99694446dc8c44a0a1e1a56ae06260fb" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 45193 } } peers { permanent_uuid: "ba35da5abc154f0e84d358b48a167607" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33439 } } }
I20260619 14:51:32.757247 23277 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba35da5abc154f0e84d358b48a167607 [sys.catalog]: This master's current role is: FOLLOWER
I20260619 14:51:32.758162 23313 mvcc.cc:205] Tried to move back new op lower bound from 7298583317357260800 to 7298583317248483328. Current Snapshot: MvccSnapshot[applied={T|T < 7298583317357260800}]
I20260619 14:51:32.767917 23308 catalog_manager.cc:1347] Generated new cluster ID: f87c18561e9f43c98d56f290cc045968
I20260619 14:51:32.768038 23308 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260619 14:51:32.819936 23308 catalog_manager.cc:1370] Generated new certificate authority record
I20260619 14:51:32.821497 23308 catalog_manager.cc:1504] Loading token signing keys...
I20260619 14:51:32.838773 23308 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a7e69f99b4a94fcb8376d048395019ba: Generated new TSK 0
I20260619 14:51:32.839699 23308 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260619 14:51:32.879204 23259 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:51:32.875334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46518 (local address 127.0.98.60:33439)
0619 14:51:32.875482 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:32.875487 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:32.877675 (+ 2188us) server_negotiation.cc:407] Connection header received
0619 14:51:32.878233 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:32.878237 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:32.878292 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:32.878349 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:32.879052 (+ 703us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46518: BlockingRecv error: recv got EOF from 127.0.0.1:46518 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260619 14:51:32.880888 23107 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:51:32.876543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47562 (local address 127.0.98.62:36421)
0619 14:51:32.876712 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:32.876717 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:32.879776 (+ 3059us) server_negotiation.cc:407] Connection header received
0619 14:51:32.880010 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:32.880023 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:32.880080 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:32.880139 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:32.880741 (+ 602us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47562: BlockingRecv error: recv got EOF from 127.0.0.1:47562 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260619 14:51:32.882333 23184 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:51:32.877299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54430 (local address 127.0.98.61:45193)
0619 14:51:32.877418 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:32.877423 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:32.881311 (+ 3888us) server_negotiation.cc:407] Connection header received
0619 14:51:32.881591 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:32.881594 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:32.881648 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:32.881703 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:32.882216 (+ 513us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54430: BlockingRecv error: recv got EOF from 127.0.0.1:54430 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260619 14:51:32.887137 23184 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:51:32.884785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54432 (local address 127.0.98.61:45193)
0619 14:51:32.884907 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:32.884911 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:32.886112 (+ 1201us) server_negotiation.cc:407] Connection header received
0619 14:51:32.886246 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:32.886249 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:32.886338 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:32.886454 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:32.887008 (+ 554us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54432: BlockingRecv error: recv got EOF from 127.0.0.1:54432 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260619 14:51:32.892091 23184 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:51:32.890651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54442 (local address 127.0.98.61:45193)
0619 14:51:32.890788 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:32.890792 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:32.890934 (+ 142us) server_negotiation.cc:407] Connection header received
0619 14:51:32.891145 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:32.891149 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:32.891204 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:32.891258 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:32.891967 (+ 709us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54442: BlockingRecv error: recv got EOF from 127.0.0.1:54442 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260619 14:51:32.902184 23184 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:51:32.896774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54444 (local address 127.0.98.61:45193)
0619 14:51:32.896908 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:32.896913 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:32.901127 (+ 4214us) server_negotiation.cc:407] Connection header received
0619 14:51:32.901298 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:32.901301 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:32.901366 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:32.901470 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:32.902052 (+ 582us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54444: BlockingRecv error: recv got EOF from 127.0.0.1:54444 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260619 14:51:32.910303 23184 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:51:32.908938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54448 (local address 127.0.98.61:45193)
0619 14:51:32.909133 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:32.909138 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:32.909325 (+ 187us) server_negotiation.cc:407] Connection header received
0619 14:51:32.909525 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:32.909528 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:32.909586 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:32.909684 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:32.910184 (+ 500us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54448: BlockingRecv error: recv got EOF from 127.0.0.1:54448 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
W20260619 14:51:32.922410 23184 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:51:32.921157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54464 (local address 127.0.98.61:45193)
0619 14:51:32.921410 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:32.921415 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:32.921432 (+ 17us) server_negotiation.cc:407] Connection header received
0619 14:51:32.921545 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:32.921548 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:32.921595 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:32.921675 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:32.922203 (+ 528us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54464: BlockingRecv error: recv got EOF from 127.0.0.1:54464 (error 108)
Metrics: {"server-negotiator.queue_time_us":165}
W20260619 14:51:32.941874 23184 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:51:32.940536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54476 (local address 127.0.98.61:45193)
0619 14:51:32.940781 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:32.940787 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:32.940923 (+ 136us) server_negotiation.cc:407] Connection header received
0619 14:51:32.941104 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:32.941108 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:32.941166 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:32.941259 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:32.941753 (+ 494us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54476: BlockingRecv error: recv got EOF from 127.0.0.1:54476 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
W20260619 14:51:32.977942 23184 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:51:32.976485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54482 (local address 127.0.98.61:45193)
0619 14:51:32.976716 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:32.976721 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:32.976745 (+ 24us) server_negotiation.cc:407] Connection header received
0619 14:51:32.977058 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:32.977061 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:32.977114 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:32.977205 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:32.977803 (+ 598us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54482: BlockingRecv error: recv got EOF from 127.0.0.1:54482 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
W20260619 14:51:33.045881 23184 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:51:33.044676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54486 (local address 127.0.98.61:45193)
0619 14:51:33.044845 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:33.044849 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:33.044923 (+ 74us) server_negotiation.cc:407] Connection header received
0619 14:51:33.045072 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:33.045075 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:33.045125 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:33.045206 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:33.045749 (+ 543us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54486: BlockingRecv error: recv got EOF from 127.0.0.1:54486 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260619 14:51:33.177879 23184 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:51:33.176193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54500 (local address 127.0.98.61:45193)
0619 14:51:33.176462 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:33.176468 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:33.176630 (+ 162us) server_negotiation.cc:407] Connection header received
0619 14:51:33.177022 (+ 392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:33.177026 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:33.177091 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:33.177198 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:33.177738 (+ 540us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54500: BlockingRecv error: recv got EOF from 127.0.0.1:54500 (error 108)
Metrics: {"server-negotiator.queue_time_us":146}
W20260619 14:51:33.438472 23184 negotiation.cc:337] Failed RPC negotiation. Trace:
0619 14:51:33.436729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54504 (local address 127.0.98.61:45193)
0619 14:51:33.437033 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:33.437037 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:33.437055 (+ 18us) server_negotiation.cc:407] Connection header received
0619 14:51:33.437149 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:33.437154 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:33.437234 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:33.437350 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0619 14:51:33.438217 (+ 867us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54504: BlockingRecv error: recv got EOF from 127.0.0.1:54504 (error 108)
Metrics: {"server-negotiator.queue_time_us":214}
I20260619 14:51:33.650923 23272 catalog_manager.cc:1259] Loaded cluster ID: f87c18561e9f43c98d56f290cc045968
I20260619 14:51:33.651047 23272 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 99694446dc8c44a0a1e1a56ae06260fb: loading cluster ID for follower catalog manager: success
I20260619 14:51:33.653141 23272 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 99694446dc8c44a0a1e1a56ae06260fb: acquiring CA information for follower catalog manager: success
I20260619 14:51:33.654992 23272 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 99694446dc8c44a0a1e1a56ae06260fb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 14:51:33.668377 23299 catalog_manager.cc:1259] Loaded cluster ID: f87c18561e9f43c98d56f290cc045968
I20260619 14:51:33.668471 23299 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ba35da5abc154f0e84d358b48a167607: loading cluster ID for follower catalog manager: success
I20260619 14:51:33.670370 23299 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ba35da5abc154f0e84d358b48a167607: acquiring CA information for follower catalog manager: success
I20260619 14:51:33.671620 23299 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ba35da5abc154f0e84d358b48a167607: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260619 14:51:33.958983 23384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:33.953066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54506 (local address 127.0.98.61:45193)
0619 14:51:33.953478 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:33.953482 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:33.953502 (+ 20us) server_negotiation.cc:407] Connection header received
0619 14:51:33.953561 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:33.953563 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:33.953703 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:33.953840 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:51:33.954335 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:33.955216 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:33.956518 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:33.957235 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:33.957360 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:33.957848 (+ 488us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:51:33.958384 (+ 536us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:51:33.958507 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:33.958634 (+ 127us) server_negotiation.cc:299] Negotiation successful
0619 14:51:33.958711 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":81,"threads_started":1}
I20260619 14:51:33.969617 23390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:33.963172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46534 (local address 127.0.98.60:33439)
0619 14:51:33.963532 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:33.963536 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:33.963554 (+ 18us) server_negotiation.cc:407] Connection header received
0619 14:51:33.963641 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:33.963644 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:33.963766 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:33.963879 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:51:33.964505 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:33.965619 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:33.967372 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:33.967688 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:33.967801 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:33.968371 (+ 570us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:51:33.969002 (+ 631us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:51:33.969127 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:33.969285 (+ 158us) server_negotiation.cc:299] Negotiation successful
0619 14:51:33.969376 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":115,"threads_started":1}
I20260619 14:51:33.978416 23390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:33.973799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46544 (local address 127.0.98.60:33439)
0619 14:51:33.973937 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:33.973942 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:33.974242 (+ 300us) server_negotiation.cc:407] Connection header received
0619 14:51:33.974626 (+ 384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:33.974631 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:33.974695 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:33.974747 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:51:33.975194 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:33.975837 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:33.977097 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:33.977460 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:33.977563 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:33.977846 (+ 283us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:51:33.977949 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:51:33.978026 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:33.978169 (+ 143us) server_negotiation.cc:299] Negotiation successful
0619 14:51:33.978248 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260619 14:51:33.978878 23384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:33.974296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54512 (local address 127.0.98.61:45193)
0619 14:51:33.974502 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:33.974506 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:33.974634 (+ 128us) server_negotiation.cc:407] Connection header received
0619 14:51:33.974759 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:33.974762 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:33.974816 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:33.974920 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:51:33.975234 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:33.976087 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:33.977431 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:33.977815 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:33.977895 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:33.978323 (+ 428us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:51:33.978404 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:51:33.978453 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:33.978652 (+ 199us) server_negotiation.cc:299] Negotiation successful
0619 14:51:33.978728 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260619 14:51:33.979221 23397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0619 14:51:33.973786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47568 (local address 127.0.98.62:36421)
0619 14:51:33.974136 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0619 14:51:33.974141 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0619 14:51:33.974580 (+ 439us) server_negotiation.cc:407] Connection header received
0619 14:51:33.974809 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0619 14:51:33.974812 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0619 14:51:33.975002 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0619 14:51:33.975141 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0619 14:51:33.975385 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:33.976208 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:33.977418 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0619 14:51:33.977767 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0619 14:51:33.977876 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0619 14:51:33.978253 (+ 377us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0619 14:51:33.978611 (+ 358us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0619 14:51:33.978700 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0619 14:51:33.979003 (+ 303us) server_negotiation.cc:299] Negotiation successful
0619 14:51:33.979078 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":122,"threads_started":1}
I20260619 14:51:33.992236 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 23036
I20260619 14:51:33.998932 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 23111
I20260619 14:51:34.005676 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskYPr0M2/build/debug/bin/kudu with pid 23185
2026-06-19T14:51:34Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3387 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6791 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (310860 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST